One such basic area in computer science is graph theory, a specific subproblem of which—the pathfinder algorithm—we’ll address in this tutorial. So first things first:

What Is a Graph?

You may already know data structures like lists, sets, and dictionaries. These data structures are denoted as complex data structures–not because they’re difficult to understand but because they build upon other data structures.

A graph is just another complex data structure for relational data.

Relational data consists of edges and vertices. Each vertex stands in one or more relations with other vertices.

An example of relational data is the Facebook social graph. Facebook represents users as vertices and friendship relations as edges. Two users are connected via an edge in the graph if they are (Facebook) friends.

What is a graph? A graph is a basic data structure in computer science. It models relationships between data items. Using graphs to model real-world phenomena is not a new idea. In 1736, Leonhard Euler has invented the graph data structure to solve the problem of “seven bridges of Königsberg”. Graphs existed way before the first computer was even an idea. In fact, as we will see in this article, graphs helped to make the computer possible. Without graphs, there wouldn’t be a computer as we now know it today.

How to Represent a Graph Data Structure in the Code?

In this tutorial, we’ll use an adjacency matrix as a graph data structure G.

Each row i in the matrix stores the out-neighbors of vertex i. And each column j stores the in-neighbors of vertex j.

Thus, there is an edge from vertex i to vertex j, if G[i][j]==1.

You can see an example of the adjacency matrix graph representation in the following code of the Pathfinder algorithm:

The Pathfinder Algorithm in Python

How to determine whether there is a path between two vertices?

The function find_path(graph, v_start, v_end, path_len) checks whether there is a direct or indirect path between two vertices v_start and v_end in graph. We know that there is a direct path between v_start and v_end if both are already neighbors, i.e., graph[v_start][v_end]==1.

def find_path(graph, v_start, v_end, path_len=0):
    '''Is there a path between vertex v_start and vertex v_end?'''

    # Traverse each vertex only once
    if path_len >= len(graph):
        return False

    # Direct path from v_start to v_end?
    if graph[v_start][v_end]:
        return True

    # Indirect path via neighbor v_nbor?
    for v_nbor, edge in enumerate(graph[v_start]):
        if edge:
            # between v_start and v_nbor
            if find_path(graph, v_nbor, v_end, path_len + 1):
                return True

    # No direct or indirect path found
    return False

# The graph represented as adjancy matrix
G = [[1, 1, 0, 0, 0],
     [0, 1, 0, 0, 0],
     [0, 0, 1, 0, 0],
     [0, 1, 1, 1, 0],
     [1, 0, 0, 1, 1]]

print(find_path(graph=G, v_start=3, v_end=0))
# False

print(find_path(G, 3, 1))
# True

However, even if there is not a direct path, there could be an indirect path between vertices v_start and v_end.

To check this, the algorithm uses a recursive approach. Specifically, there is an indirect path if a vertex v_nbor exists such that there is a path:

 v_start --> v_nbor --> ... --> v_end

The variable path_len stores the length of the current path.

We increment it in each recursion level as the current path length increases by one. Note that all paths with length >=n consist of at least n vertices.

In other words, at least one vertex is visited twice and a cycle exists in this recursion instance. Hence, we skip recursion for paths with lengths greater than or equal to the number of vertices in the graph.

In the code snippet, we check whether there is a path between 3 and 0.

If you understand what the code is doing, it suffices to look at the adjacency matrix G.

There is a direct path from vertex 3 to vertices 1 and 2 (and to itself). But neither vertex 1 nor 2 has any out-neighbors.

Therefore, there is no path from vertex 3 to any other vertex (besides vertices 1 and 2).

Related Video

Academy Course – Mastering the Top 10 Graph Algorithms

If you want to improve your fundamental computer science skills, there’s nothing more effective than studying algorithms.

To help you master the most important graph algorithms, we’ve just launched the “Top 10 Algorithms” course at the Finxter Computer Science Academy. This great course from Finxter Star Creator Matija teaches you the most important graph algorithms such as BFS, DFS, A*, and Dijkstra. 

Understanding these algorithms will not only make you a better coder, but it’ll also lay a strong foundation on which you can build your whole career as a computer scientist.

Click the screenshot to find out more:

The post A Recursive Pathfinder Algorithm in Python appeared first on Be on the Right Side of Change.

Definition weighted average: Each array element has an associated weight. The weighted average is the sum of all array elements, properly weighted, divided by the sum of all weights.

Here’s the problem exemplified:

Quick solution: Before we discuss the solution in great detail, here’s the solution that solves this exact problem:

import numpy as np

array = np.array([[1, 0, 2],
                  [1, 1, 1]])
weights = np.array([[2, 1, 1],
                    [1, 1, 2]])

print(np.average(array, weights=weights))
# 1.0

Want to learn how it works—and how you can average along an axis as well? Let’s dive deeper into the problem next!

Weighted Average with NumPy’s np.average() Function

NumPy’s np.average(arr) function computes the average of all numerical values in a NumPy array. When used with only one array argument, it calculates the numerical average of all values in the array, no matter the array’s dimensionality. For example, the expression np.average([[1,2],[2,3]]) results in the average value (1+2+2+3)/4 = 2.0.

However, what if you want to calculate the weighted average of a NumPy array? In other words, you want to overweight some array values and underweight others.

You can easily accomplish this with NumPy’s average function by passing the weights argument to the NumPy average function.

import numpy as np

a = [-1, 1, 2, 2]

print(np.average(a))
# 1.0

print(np.average(a, weights = [1, 1, 1, 5]))
# 1.5

In the first example, we simply averaged over all array values: (-1+1+2+2)/4 = 1.0. However, in the second example, we overweight the last array element 2—it now carries five times the weight of the other elements resulting in the following computation: (-1+1+2+(2+2+2+2+2))/8 = 1.5.

NumPy Weighted Average Video

NumPy Average Syntax

Let’s explore the different parameters we can pass to np.average(...).

• The NumPy array which can be multi-dimensional.
• (Optional) The axis along which you want to average. If you don’t specify the argument, the averaging is done over the whole array.
• (Optional) The weights of each column of the specified axis. If you don’t specify the argument, the weights are assumed to be homogeneous.
• (Optional) The return value of the function. Only if you set this to True, you will get a tuple (average, weights_sum) as a result. This may help you to normalize the output. In most cases, you can skip this argument.

average(a, axis=None, weights=None, returned=False)

NumPy Weighted Average Along an Axis (Puzzle)

Here is an example how to average along the columns of a 2D NumPy array with specified weights for both rows.

import numpy as np

# daily stock prices
# [morning, midday, evening]
solar_x = np.array(
    [[2, 3, 4], # today
     [2, 2, 5]]) # yesterday

# midday - weighted average
print(np.average(solar_x, axis=0, weights=[3/4, 1/4])[1])

What is the output of this puzzle?
*Beginner Level* (solution below)

You can also solve this puzzle in our puzzle-based learning app (100% FREE): Test your skills now!

Puzzle Explanation

Numpy is a popular Python library for data science focusing on arrays, vectors, and matrices.

This puzzle introduces the average function from the NumPy library. When applied to a 1D NumPy array, this function returns the average of the array values. When applied to a 2D NumPy array, it simply flattens the array. The result is the average of the flattened 1D array.

In the puzzle, we have a matrix with two rows and three columns. The matrix gives the stock prices of the solar_x stock. Each row represents the prices for one day. The first column specifies the morning price, the second the midday price, and the third the evening price.

Now suppose, we do not want to know the average of the flattened matrix but the average of the price in the midday. Moreover, we want to overweight the most recent stock price. Today accounts for three-quarters and yesterday for one-quarter of the final average value.

NumPy enables this via the weights parameter in combination with the axis parameter.

• The weights parameter defines the weight for each value participating in the average calculation.
• The axis parameter specifies the direction along which the average should be calculated.

In a 2D matrix, the row is specified as axis=0 and the column as axis=1. We want to know three average values, for the morning, midday, and evening. We calculate the average along the row, i.e., axis=0. This results in three average values. Now we take the second element to get the midday variance.

Recommended: How to Normalize a NumPy Matrix

Where to Go From Here?

Want to thrive in data science? Master NumPy first. You need to know the most important concepts (such as the axis argument) before you dive into machine learning and data science. Only then, you can properly understand the algorithms and build your career on a solid foundation.

To help you accomplish this, we’ve written an easy-to-read, fun introduction into the NumPy library. It’s 100% based on puzzle-based learning: you solve rated NumPy puzzles, test your skills, and improve over time. Check it out—it’s fun!

Solution

2.75

The post How to Calculate the Weighted Average of a Numpy Array in Python? appeared first on Be on the Right Side of Change.

When applied to a 1D array, this function returns the average of the array values. When applied to a 2D array, NumPy simply flattens the array. The result is the average of the flattened 1D array. Only if you use the optional axis argument, you can average along the rows or columns of the 2D array.

Here’s a visual overview first—we’ll discuss details later:

Let’s start with the simple, flat case first.

Average of Flattened 2D Array

To calculate the average of all values in a two-dimensional NumPy array called matrix, use the np.average(matrix) function.

>>> import numpy as np
>>> matrix = np.array([[1, 0, 2],
                       [1, 1, 1]])
>>> np.average(matrix)
1.0

This calculates the average of the flattened out matrix, i.e., it’s the same as calling np.average([1, 0, 2, 1, 1, 1]) without the two-dimensional structuring of the data.

Column Average of 2D Array

To calculate the average separately for each column of the 2D array, use the function call np.average(matrix, axis=0) setting the axis argument to 0.

>>> np.average(matrix, axis=0)
array([1. , 0.5, 1.5])

The resulting array has three average values, one per column of the input matrix.

Row Average of 2D Array

To calculate the average separately for each row of the 2D array, call np.average(matrix, axis=1) setting the axis argument to 1.

>>> np.average(matrix, axis=1)
array([1., 1.])

The resulting array has two average values, one per row of the input matrix.

NumPy Puzzle Average

To test your skills and train your understanding of the np.average() function, here’s a code puzzle you may enjoy:

import numpy as np

# stock prices (3x per day)
# [morning, midday, evening]
solar_x = np.array(
    [[2, 3, 4], # day 1
     [2, 2, 5]]) # day 2

print(np.average(solar_x))

What is the output of this puzzle?
*Beginner Level* (solution below)

You can solve this code puzzle interactively on our Finxter.com puzzle app here:

The puzzle has a matrix with two rows and three columns. The matrix gives the stock prices of the solar_x stock. Each row represents the prices for one day. The first column specifies the morning price, the second the midday price, and the third the evening price.

Note that NumPy calculates the average as the sum of all values divided by the number of values. The result is a float value.

Are you a master coder?
Test your skills now!

Note that the following tutorial may be interesting to you too:

Recommended Tutorial: How to Get the Length of a 2D NumPy Array

Related Video

The post How to Calculate the Average of a NumPy 2D Array? appeared first on Be on the Right Side of Change.

Problem Formulation: How to index elements in NumPy arrays?

Indexing 1D Arrays with Positive Indices

The most simple use of indexing is with the square bracket notation and positive integers:

>>> import numpy as np
>>> a = np.array([1, 2, 3])
>>> a
array([1, 2, 3])
>>> a[0]
1
>>> a[1]
2
>>> a[2]
3

If you use a positive index larger or equal than the number of elements in the array, Python will throw an IndexError:

>>> import numpy as np
>>> a = np.array([1, 2, 3])
>>> a[3]
Traceback (most recent call last):
  File "<pyshell#19>", line 1, in <module>
    a[3]
IndexError: index 3 is out of bounds for axis 0 with size 3

Indexing 1D Arrays with Negative Indices

You can also use negative indices to access the array elements, starting with the last element and moving to the left:

>>> import numpy as np
>>> a = np.array([8, 7, 5, 4, 9, 1, 9, 5])
>>> a[-1]
5
>>> a[-2]
9
>>> a[-3]
1
>>> a[-4]
9
>>> a[-5]
4
>>> a[-6]
5
>>> a[-7]
7
>>> a[-8]
8

If you move further into the negative, Python will throw an IndexError:

>>> a[-9]
Traceback (most recent call last):
  File "<pyshell#17>", line 1, in <module>
    a[-9]
IndexError: index -9 is out of bounds for axis 0 with size 8

Indexing 2D Arrays NumPy

If you use two-dimensional arrays, you can index individual elements with the square bracket notation and comma-separated index values, one per axis. The first index value gives the row index and the second index value gives the column index:

>>> import numpy as np
>>> a = np.array([[42, 8, 7],
		  [99, 3, 4]])
>>> a[0, 0]
42
>>> a[1, 2]
4
>>> a[1, 1]
3

You can also use negative indexing on one or both axes.

>>> a[-1, -1]
4

If you access elements outside the bound of the maximal possible index, Python raises an IndexError:

>>> a[2, 1]
Traceback (most recent call last):
  File "<pyshell#28>", line 1, in <module>
    a[2, 1]
IndexError: index 2 is out of bounds for axis 0 with size 2

NumPy Array Indexing Multi-dimensional Arrays

If you use multi-dimensional arrays, you can index individual elements with the square bracket notation and comma-separated index values, one per axis.

>>> import numpy as np
>>> a = [[[1, 1], [2, 3]],
	 [[4, 3], [8, 9]]]
>>> a = np.array(a)
>>> a[0, 0, 0]
1
>>> a[0, 0, 1]
1
>>> a[0, 1, 0]
2
>>> a[0, 1, 1]
3
>>> a[1, 0, 0]
4
>>> a[1, 0, 1]
3
>>> a[1, 1, 0]
8
>>> a[1, 1, 1]
9

As a rule of thumb: the first element in the comma-separated square bracket notation identifies the outermost axis, the second element the second-outermost axis, and so on.

NumPy Array Indexing Puzzle

Train your skills by solving the following NumPy puzzle about indexing and basic array arithmetic:

import numpy as np

# air quality index AQI data
hong_kong = np.array([42, 40, 41, 43, 44, 43])
new_york = np.array([30, 31, 29, 29, 29, 30])
montreal = np.array([11, 11, 12, 13, 11, 12])

hk_mean = (hong_kong[0] + hong_kong[-1]) / 2.0
ny_mean = (new_york[1] + new_york[-3]) / 2.0
m_mean = (montreal[1] + montreal[-0]) / 2.0

print(hk_mean)
print(ny_mean)
print(m_mean)

What is the output of this puzzle?
*Beginner Level* (solution below)

You can solve this puzzle on our interactive Finxter app and track your skill level here:

The puzzle analysis data from the real-time air quality index (AQI) for the three cities Hong Kong, New York, and Montreal. The index data aggregates various factors that influence the air quality such as respirable particulate matter, ozone, and nitrogen dioxide. The goal is to compare the air quality data for the three cities. To show how indexing works, we use different indexing schemes to access two data values for each city. Then, we normalize the data by 2.0.

You can use positive or negative indices. For positive indices, use 0 to access the first element and increment the index by 1 to index each subsequent element. For negative indices, use -1 to access the last element and decrement the index by 1 to access each previous element. It’s as simple as that.

Are you a master coder?
Test your skills now!

Related Video

The post How to Index Elements in NumPy Arrays? appeared first on Be on the Right Side of Change.

If you use Netflix or Amazon you have already seen the results of recommendation systems – movie or item recommendations that fit your taste or needs. So, at its core a recommendation system is a statistical algorithm that computes similarities based on previous choices or features and recommends users which movie to watch or what else they might need to buy.

How Does a Recommendation System Work?

Assume that persons A and B like a movie M1 and person A also likes movie M2. Now, we can conclude that person B will also like movie M2 with a high probability. Well, that’s very little data and probably a rather imprecise prediction. Yet, it illustrates how collaborative filtering works. In a real world application we would need much more data to make good recommendations. The recommendation algorithms based this concept are called collaborative filtering.

Another popular way to recommend items is so called content-based filtering. Content-based filtering computes recommendations based on similarities of items or movies. In the case of movies we could look at different features like: genre, actors, … to compute similarity.
If a user liked a given movie, the probability is high that the user will also like similar movies. Thus, it makes sense to recommend movies with a high similarity to those the user liked.

Implementing a Recommendation System

If you want to understand the code below better, make sure to sign up for our free email course “Introduction to Pandas and Data Science” on our Email Academy. Throughout the course, we develop a recommendation system for movies. At its core, there is the method corrwith() from the Pandas library.

This is the final implementation of our recommendation system:

How to Use Pandas corrwith() Method?

The Pandas object DataFrame offers the method corrwith() which computes pairwise correlations between DataFrames or a DataFrame and a Series. With the parameter axis, you can either compute correlations along the rows or columns. Here is the complete signature, blue parameters are optional and have default values.

The arguments in detail:
1.) other: A Series or DataFrame with which to compute the correlation.
2.) axis: Pass 0 or ‘index’ to compute correlations column-wise, 1 or ‘columns’ for row-wise.
3.) drop: Drop missing indices from result.
4.) method: The algorithm used to compute the correlation. You can either choose from: ‘pearson’, ‘kendall’ or ‘spearman’ or implement your own algorithm. So, either you pass one of the three strings or a callable.

Here is a practical example:

import pandas as pd

ratings = {
       'Spider Man':[3.5, 1.0, 4.5, 5.0],
       'James Bond':[1.0, 2.5, 5.0, 4.0],
       'Titanic':[5.0, 4.5, 1.0, 2.0] 
}

new_movie_ratings = pd.Series([2.0, 2.5, 5.0, 3.5])
all_ratings = pd.DataFrame(ratings)

print(all_ratings.corrwith(new_movie_ratings))

From a given dictionary of lists (ratings) we create a DataFrame. This DataFrame has three columns and four rows. Each column contains the movie ratings of all four users.
The Series new_movie_ratings contains the ratings for a new movie of all four users.
Using the method corrwith() on the DataFrame we get the correlation between the new ratings and the old ones.
The output of the snippet above is:

Spider Man    0.566394
James Bond    0.953910
Titanic      -0.962312

As you can see, the new movie has the highest correlation with the James Bond movie. This means, a recommendation system which works purely based on ratings, should recommend the James Bond movie to users that liked the new movie.
Yet, what exactly is correlation?

What is Correlation?

Correlation describes the statistical relationship between two entities. This is to say, it’s how two variables move in relation to one another. Correlation is given as a value between -1 and +1. However, correlation is not causation!

There are three types of correlation:

• Positive correlation:
  A positive correlation is a value in the range 0.0 < c <= 1.0. A correlation of 1.0 means that if the first variable moves up, the second one will also move up. This relationship is weaker if the correlation is lower than 1.0.
• Negative correlation:
  A negative correlation is a value in the range 0.0 > c >= -1.0. Negative correlation means that two variable have the opposite behaviour. So, if the first one moves up the second one moves down.
• Zero or no correlation:
  A correlation of zero means there is no relationship between the two variables. If the first variable moves up, the second one may do anything else.

More Pandas DataFrame Methods

Feel free to learn more about the previous and next pandas DataFrame methods (alphabetically) here:

df.corr()

df.count()

Also, check out the full cheat sheet overview of all Pandas DataFrame methods.

The post A Simple Recommendation System Using Pandas corrwith() Method appeared first on Be on the Right Side of Change.

1. Motivation: Watch the free video about the state-of-the-art of Python freelancing (opens in a new tab).
2. Training: Reach Python freelancer level (~40-80 hours).
3. Confidence: Complete 3 archived Python projects for learning and read business books such as Leaving the Rat Race with Python (Amazon).
4. Platform: Create accounts at Upwork and Fiverr.
5. Credibility: Get your 5-star ratings for small $15 Python projects.
6. Scale: Ladder up your hourly wage.
7. Pro tip: Create a personal website and drive paid traffic to it.

Do you want to earn money as a Python freelancer? But you are just starting in Python? This article leads you step-by-step through the adventure of becoming a Python freelancer. You’ll learn about the exact steps you need to do to become a Python freelancer—starting as a Python newbie.

And if you’re serious about your next-level career step, join my FREE webinar (on the Finxter blog) “How to Build Your High-Income Skill Python” where I’ll show you exactly how I went from earning $0 to full-time income and beyond—as a Python freelance developer.

Before we start, let’s answer an important question: is Python freelancing worth it? I compiled the pros and cons from different places online (e.g. here) and added my own experience in the freelancing space—both as a client and as a Python freelancer.

Do you want to develop the skills of a well-rounded Python professional—while getting paid in the process? Become a Python freelancer and order your book Leaving the Rat Race with Python on Amazon (Kindle/Print)!

Without losing any time, let’s dive into the 7 steps of becoming a Python freelancer.

Table of contents:

1. How much money ($$$) you can earn as a Python freelancer
2. How to gain confidence that you can give value to the marketplace?
3. How to start learning the basics of Python?
4. How to make yourself independent from freelancing platforms?
5. How to become an attractive freelancer?
6. How to find practical Python projects for learning?
7. When to stop learning and start doing real Python projects?
8. What is a good place to start Python freelancing?

1. How Much Money ($$$) Can You Earn as a Python Freelancer?

As a Python developer, you can expect to earn between $10 and $80 per hour with an average salary of $51 (source). I know the variation of the earning potential is high but so is the quality of the Python freelancers in the wild. Take the average salary as a starting point and add +/- 50% to account for your level of expertise. In this blog article, I summarized all credible sources online about how much you can earn as a Python coder. The key takeaway is that intermediate-level Python freelancers today earn six figures easily ($100,000 yearly gross income or more):

This data is based on various sources:

• Codementor argues that the average freelancer earns between $61 and $80 in 2019: source
• This Subreddit gives a few insights about what some random freelancers earn per hour (it’s usually more than $30 per hour): source
• Ziprecruiter finds that the average Python freelancer earns $52 per hour in the US. This is equivalent to $8,980 per month or $107,000 per year: source
• Payscale is more pessimistic and estimates the average hourly rate around $29 per hour: source
• As a Python developer, you can expect to earn between $10 and $80 per hour with an average salary of $51 (source).

If you work on the side, let’s make it 8 hours each Saturday, you will earn $400 extra per week – or $1600 per month (before taxes). Your hourly rate will be a bit lower because you have to invest time finding freelancing clients – up to 20% of your total time.

Action steps:

• Write down how many hours you can invest per week.
• Write down your goal hourly rate.

2. How to Gain Confidence That You Can Give Value to the Marketplace?

Before becoming a Python freelancer, you have to learn the very basics of Python. What’s the point of offering your freelancer services when you can not even write Python code?

Having said this, it’s more likely that you live on the other extreme. You do not want to offer your services before you don’t feel 100% confident about your skills. Unfortunately, this moment never arrives. I have met hundreds of advanced coders, who are still not confident in selling their services. They cannot overcome their self-woven system of limiting believes and mental barriers.

Can I tell you a harsh truth? You won’t join the top 1% of the Python coders with high probability (a hard statistical fact). But never mind. Your services will still be valuable to clients who either have less programming skills (there are plenty of them) or little time (a big part of the rest).  Most clients are happy to outsource the complex coding work to focus on their key result areas.

Regardless of your skill level, the variety of Python projects is huge. There are simple projects for $10 which an experienced coder can solve in 5 minutes. And there are complex projects that take months and promise you large payments of $100 to $1000 after completing each milestone.

You can be sure that you will find projects in your skill level.

Action steps:

• Take your time to browse all the archived Python freelance projects.
• Select 3 projects that you think you can solve in the price range ($10-$50).
• Write down in which direction you want to go first (keep the projects in mind that you just selected): data science, web scraping, application development, scripting, …

3. How to Start Learning the Basics of Python?

Before you start with practical projects though, you should invest 10-20 hours in refreshing your basic Python skills. This is not much of a time commitment – after all, you are learning a high-income skill. You can learn a lot in 20 hours if you do it right. The key is to learn probabilistically by mastering important subskills first. Watch this great TED talk about what you can achieve in 20 hours.

So what’s the best way to learn probabilistically? Simple. Use the 80/20 principle. This famous principle states that 80% of the causes lead to 20% of the effects. Get rid of the 80% low-value tasks and focus instead of the 20% of causes with 80% of the effects.

The best way to learn 80% of the skills in 20% of the time, is via Python cheat sheets. I have summarized the 5 best Python cheat sheet in this article. Download the cheat sheets and spend your first 20 hours in learning them thoroughly. Or even better: print them and post them to your office wall.

Action steps:

• Download the cheat sheet: Python 3 Cheat Sheet.
• Subscribe to the FREE cheat sheet course: The Ultimate Python Cheat Sheet Course (5x Email Series).

4. How to Gain Independence from Freelancing Platforms?

Freelancing platforms offer you convenience and speed in starting your own freelancing business. Examples are Upwork, Freelancer, or even Fiverr. Some of those platforms (e.g. Upwork) will manually check your profile, so it will take some time to sign up. The reason is that these platforms receive thousands of applications every day. They have to be selective to guarantee a certain quality of service. I have read about many cases where Upwork rejected freelancers with attractive profiles.

On the first impression, these platforms seem to be very attractive for your emerging freelance business. But be careful! Many existing freelancers heavily rely on these freelancing platforms. By using a platform such as Upwork, you make yourself vulnerable. Your income depends on the decisions of the platform owners. And don’t think they have your best interest at heart. For example, Upwork already takes a 20% cut (!) from your hourly rate, just for brokering your services to potential clients. And their cut is pre-tax. In other words, you are working the first 12 minutes of each hour for Upwork (and the next 28 minutes for the government).

What are some risks of depending on a freelancing platform? The platform owners can suddenly increase their cut. They can throw you from the platform for no reason whatsoever. They will stop sending clients your way as soon as their algorithm decides that you are not the optimal fit for a client project. The main problem is that you don’t control your customer base. On the back of ONLY these platforms, you can not build a robust and sustainable business.

So what’s the alternative? If you are serious about becoming a Python freelancer, setting up your own website is the way to go. This allows you to offer your services to clients all over the world. You establish trust and clients perceive you as a professional freelancer. Use freelancing platforms to attract clients, but retain them for yourself.

To increase your conversions, add testimonials to your website. Keep collecting them during each of your freelancing activities. Of course, this is a bit harder than just setting up a freelancing account on Upwork. But it’s a much more sustainable foundation of your freelancing business. A nice plus is that your professional website sets you apart from your competition. It increases your chances of getting clients.

Only after creating your own web presence, you should use these platforms to get new clients. Make sure to always refer to your professional website within any project application.

How do you retain clients beyond the first project? Focus on always over-delivering to new clients! Make them happy. Ask them to contact you directly the next time they need a similar service. And give them a special offer for the next freelancing service. Finally, ask them for referrals and testimonials after the job is done.

As you establish a growing client base you control, you will find yourself using Upwork, Freelancer, and Fiverr less and less. Because of that, your profit margin will grow over time. Not only will your income increase, but it will also stand on a solid foundation. You own the foundation of your business (your client base).

Action steps:

• Get a meaningful domain such as (e.g. “python-freelancing-services.com”).
• Create a WordPress page introducing your services.
• Create profiles on Freelancer.com, Upwork.com, and Fiverr.com.

5. How to Become an Attractive Freelancer?

In the last months, I got a lot of experience using freelancer services as a client. My goal was to improve the Finxter.com website to test your Python skills and my book “Coffee Break Python”. A similar pattern emerged every time I posted a new project description. A few hours after posting the description, several freelancers applied. The competition was fierce. But within minutes, I had subconsciously chosen an inner circle of high-potential candidates. There was not one case, where I chose a freelancer who could not immediately pass the entry barriers of my subconscious mind.

You want to be in that inner circle. To get there, you must be appealing to the subconscious mind. The following factors will give you a psychological advantage when competing for a freelance job.

Use the power of reciprocity.

“Reciprocity is a social norm of responding to a positive action with another positive action, rewarding kind actions.”

Wikipedia

This principle is the most important insight which I gained from my own experience of hiring freelancers.

For example, I published a project to check the Python code of my recent Python programming book. The project description stated that I intended to edit the code to make it more Pythonic. This project was important to me because I want to provide high quality and readability for advanced coders. Immediately, several freelancers applied for the job. They went into “competition-mode” bragging about their credentials. They tried to convince me that they are the perfect fit for this project. I selected a few candidates but was not 100% sure about any of them.

Towards the end of the application phase, a new freelancer registered interest with an unusual application. Instead of talking about his credentials, he focused on the project itself. He dived right into the project and submitted annotated and corrected Python code snippets – improving upon those I provided as sample files. He gave them to me for free. Of course, I knew that he purposely used the reciprocity rule to get the job. Yet, I was immediately hooked and felt a strong obligation to reward him for his work – and gave him the job.

This is the power of the reciprocity rule.  

Don’t hide your titles and credentials.

They still work. When you apply for a job and you have the title “Prof.”, “Ph.D.”, or “MSc” in a relevant area, you have gained immediate credibility. In most cases, it will set you apart from the other freelancers without strong credentials or titles.

Note that credentials are not limited to the academic world. You should also highlight your practical achievements such as your websites, shiny projects, or certificates. Be creative.

Invest time in your profile picture.

You wouldn’t believe the powerful impact of your profile picture on your chances of getting the job. Many coders don’t focus too much on appearance. Don’t do this. Smile, dress professionally, use a natural image background. Need insider tips? Check out this article from freelancer.com.

Don’t compete on price.

Forget about it. Competing on price is a race towards zero. You can not win. There is always a cheaper freelancer and some of them WILL apply for the same projects.

It’s true, some clients look for the cheapest freelancer who barely finishes the task. But most clients will choose high quality and predictability over price. What would you do if you were a business owner who works 60 hour weeks to push his website? You love your baby and don’t want a cheap freelancer to mess around with it. A freelancer that offers a service at a very cheap rate is also perceived as shipping cheap quality. After all, you ain’t cheap if you are good.

So what is the value of an hour of your work? Multiply this number with 1.5. Do this for two reasons: You tend to underestimate your value to the marketplace, and you should constantly push yourself to improve (that’s what you want, isn’t it?). Now you have your number. NEVER work for an hourly rate below that number! And keep pushing it – the sky is the limit!

Action steps:

• Collect Python certificates. For example, use our web app Finxter to certify your Python skill level.
• Get an awesome profile picture.
• Give something to each potential client. For example, invest some time creating a prototype solution. This will greatly improve your acceptance rate and ultimately save a lot of time!

6. How to Find Practical Python Projects for Learning?

This is the most important question for you as a beginner. Most developers know that they should never do premature code optimizations. But they do premature skill optimizations all the time. Don’t do that. Laser focus your time to learn the most important skills with the highest priority.

One of my articles on this blog shows you 10 highly practical freelancer projects on which real freelancers worked to earn money. Again, find highly practical code projects for any skill level at freelancer.com/archives.

These archived freelancing projects are goldmines. Invest time to study them carefully. If you do so, you will learn about the practical Python problems that clients seek to solve. You will learn about the patterns of Python problems “in the wild”. This knowledge will guide you in your efforts of becoming more valuable to the marketplace. In contrast to millions of other aspiring coders, you will develop a practical Python skillset tailored to your interest level.

Action steps:

• Browse the archive of freelancing Python projects.
• Find projects you like in your difficulty level (read my article about 10 highly practical freelancer projects with which real freelancers earned money).
• Complete these projects.

7. When to Stop Learning and Start Doing Real Python Projects?

Start with real projects immediately. Use the freelance.com archive to find practical projects if you do not feel confident, yet. But don’t wait too long – even if you are a beginner programmer. Set aside 10 minutes per day or so to watch out for interesting projects. When you have solved archived projects, chances are that clients have similar projects that they want to outsource. To drive traffic to your own freelancer website, you could write a blog post about the topic. You could even drive traffic via Facebook ads if you need to. Try to establish your own brand as soon as possible. Learn with real clients. Learn practical skills first. What are the practical skills? You can only learn them by working on real projects. Don’t lose any more time. Start today!

My answer is very simple: start right away — no matter your current skill level.

But I know that for many Python coders just starting out, it’s very difficult to start right away. Why? Because they don’t have the confidence, yet, to start taking on projects.

And the reason is that they never have quite finished a Python project — and, of course, they are full of doubts and low self-esteem. They fear not being able to finish through with the freelancer project and earn the criticism of their clients.

If you have to overcome this fear first, then I would recommend that you start doing some archived freelancer projects. I always recommend a great resource where you can find these archived freelancer projects. On this resource, you’ll find not only a few but all the freelancer projects in different areas — such as Python, data science, and machine learning — that have ever been published at the Freelancer.com platform. There are thousands of such projects.

Unfortunately, many projects published there are crappy and it’ll take a lot of time finding suitable projects. To relieve you from this burden, I have compiled a list of 10 suitable Python projects (and published a blog article about that) which you can start doing today to improve your skill level and gain some confidence. Real freelancers have earned real money solving these projects — so they are as practical as they can be.

I recommend that you invest 70% of your learning time finishing these projects. First, you select the project. Second, you finish this project. No matter your current skill level. Even if you are a complete beginner then it will just take you weeks to finish the project which earned the freelancer 20 dollars. So what? Then you have worked weeks to earn $20 (which you would have invested for learning anyways) and you have improved your skill level a lot. But now you know you can solve the freelancer project.

The next projects will be much easier then. This time, it’ll take you not weeks but a week to finish a similar project. And the next project will take you only three days. And this is how your hourly rate increases exponentially in the beginning until you reach some convergence and your hourly rate flattens out. At this point, it’s important that you specialize even further. Select the skills that interest you and focus on those skills first. Always play your strengths.

If you need some more confidence then go read my article about the 10 practical Python freelancer projects, select one, and finish your first project yourself. This way, your learning will always be as practical as it can be.

If you want to know how much you can earn and get the overall picture of the state of Python freelancing in 2019, then check out my free webinar: How to earn $3000/M as a Python freelancer. It’ll take you only 30-40 minutes and I’ll explain you in detail the state of the art in freelancing, future outlooks and hot skills, and how much you can earn compared to employees and other professions.

As a Quora user puts it: “rank communication before development skills”. It’s critical for your success that you develop these communication skills in a practical environment.

Action step:

• Take the first project you think you could solve. Then invest all your time and effort into cracking this project. Learn on the way. A sure way to improve and build a relationship with your clients – and make money in the process.

8. What is a Good Place to Start Python Freelancing?

There are many different ways of starting your Python freelancing adventures. Many freelancing platforms compete for your time, attention, and a share of your value creation. These platforms are a great way to start your freelancing career as a Python coder and gain some experience in business and coding, as well as get some testimonial to kick off your freelancing business. But keep in mind that they are only the first step and in the mid-term, you should strive to become independent of those platforms if you want to avoid global competition for each project in the future.

So without further delay, these are the best places to start your Python freelancing career and get some clients fast (ordered by my recommendation):

Upwork

Upwork places a great focus on quality. This is great for clients because it ensures that their work will get delivered—without compromising quality.

For freelancers just starting out, Upwork poses a significant barrier of entry—oftentimes, new profiles will get rejected by the Upwork team. They want to ensure that only clients who take their freelancing jobs seriously will start out on their platform.

However, the relatively high barrier of entry also protects established freelancers on the Upwork platform from too much competition. There is no price dumping because of low-quality offers which ultimately benefits all market participants.

Fiverr

Fiverr initially started out as a platform where you could buy and sell small gigs worth five bucks. However, in the meantime, it grew to a full-fledged freelancing platform where people earn six-figure incomes.

Many jobs earn hundreds of dollars per hour and many freelancers make a killing—especially in attractive industries such as programming, machine learning, and data science.

If you want to start earning money as a freelance developer with the hot Python programming language, check out my free webinar:

How to build your high-income skill Python [Webinar]

Toptal

Toptal has a strong market proposition: it’s the platform with the top 3% of freelancers. Hence, it connects high-quality freelancers with high-quality clients.

It’s extremely hard to become a freelancer at Toptal: 97% of the applicants will not enter the platform. However, if you manage to join Toptal, you can greatly benefit from the best-in-class hourly rates. You can easily earn $100 per hour and beyond.

Also, the high barrier of entry ensures that the freelancer stays the valuable resource—he or she doesn’t become a commodity like on other freelancer platforms.

If you are an upcoming freelancer, you should aim for joining Toptal one day. Here’s a great freelancer course that shows you a crystal-clear path towards becoming a highly-paid freelancer.

Freelancer.com

Freelancer.com is the go-to resource for beginners with a very low barrier of entry and opportunities for everyone. This is the recommended starting point to gain experience and finish your first projects. Also, it can help you gain your first testimonials—while getting paid for learning and polishing your skills.

It’s a great site with countless freelancing projects. A great resource is the archived freelancing projects which help you get some real-world projects for training purposes.

For freelance programmers, I have compiled a list of ten practical freelancing projects to help you get started. These projects are real projects which were completed by real freelancers for real money. So they are as practical as they can get.

Of course, there are a lot more general freelancing websites. I will list a few of them in the following:

• https://www.peopleperhour.com/
• https://remote.com/
• https://www.guru.com/
• https://www.truelancer.com/

Also, if you are looking for specialized freelancing platforms, you should look a bit further. For example, an excellent way of offering your writing services is:

iWriter

The best site for offering your programming services is:

Twago

However, the general freelancing sites such as Upwork, Toptal, and Fiverr are good enough for most cases. To learn more about the best Python freelancer sites, check out my original blog article.

Action step:

• Before you move on, decide for one platform (my personal recommendation: Upwork — and, no, I’m not affiliated with any institution) and stick for it for at least one year. Commitment is king!

More Finxter Tutorials

Learning is a continuous process and you’d be wise to never stop learning and improving throughout your life.

What to learn? Your subconsciousness often knows better than your conscious mind what skills you need to reach the next level of success.

I recommend you read at least one tutorial per day (only 5 minutes per tutorial is enough) to make sure you never stop learning!

If you want to make sure you don’t forget your habit, feel free to join our free email academy for weekly fresh tutorials and learning reminders in your INBOX.

Also, skim the following list of tutorials and open 3 interesting ones in a new browser tab to start your new — or continue with your existing — learning habit today!

Python Basics:

• Python One Line For Loop
• Import Modules From Another Folder
• Determine Type of Python Object
• Convert String List to Int List
• Convert Int List to String List
• Convert String List to Float List
• Convert List to NumPy Array
• Append Data to JSON File
• Filter List Python
• Nested List

Python Dependency Management:

• Install PIP
• How to Check Your Python Version
• Check Pandas Version in Script
• Check Python Version Jupyter
• Check Version of Package PIP

Python Debugging:

• Catch and Print Exceptions
• List Index Out Of Range
• Fix Value Error Truth
• Cannot Import Name X Error

Fun Stuff:

• 5 Cheat Sheets Every Python Coder Needs to Own
• 10 Best Python Puzzles to Discover Your True Skill Level
• How to $1000 on the Side as a Python Freelancer

Thanks for learning with Finxter!

The post How to Become a Python Freelancer—and Earn $1,000 on the Side? [A Step-by-Step Tutorial] appeared first on Be on the Right Side of Change.

For your convenience, I’ve created a comprehensive 8000-word eBook which you can download directly as a high-resolution PDF (opens in a new window).

Unless stated otherwise I will be using Python 3.8 throughout. Dictionary functionality has changed over the last few Python versions. If you are using a version other than 3.8, you will probably get different results.

To check what version of Python you are running, enter the following in a terminal window (mine returns 3.8).

$ python --version
Python 3.8.0

Here’s a minimal example that shows how to use a dictionary in an interactive Python shell. Feel free to play around!

Exercise: Add 2 apples and 3 oranges to your basket of fruits! How many fruits are in your basket?

Python Dictionary Video Tutorial

Don’t want to read the article? No problem, watch me going over the article:

Here’s the link to the Python freelancer course in case you want to start being your own boss with Python.

Python Dictionary – Why Is It So Useful?

When I first found out about dictionaries, I wasn’t sure if they were going to be very useful. They seemed a bit clunky and I felt like lists would be much more useful. But boy was I wrong!

In real life, a dictionary is a book full of words in alphabetical order. Beside each word is a definition. If it has many meanings, there are many definitions. Each word appears exactly once.

• A book of words in alphabetical order.
• Each word has an associated definition
• If a word has many meanings, it has many definitions
• As time changes, more meanings can be added to a word.
• The spelling of a word never changes.
• Each word appears exactly once.
• Some words have the same definition.

If we abstract this idea, we can view a dictionary as a mapping from a word to its definition. Making this more abstract, a dictionary is a mapping from something we know (a word) to something we don’t (its definition). 

We apply this mapping all the time in real life: In our phone, we map our friends’ names to their phone numbers.

In our minds, we map a person’s name to their face.

We map words to their meaning.

This ‘mapping’ is really easy for humans to understand and makes life much more efficient. We do it all the time without even realising. Thus it makes sense for Python to include this as a foundational data type. 

Python Dictionary Structure

A traditional dictionary maps words to definitions. Python dictionaries can contain any data type, so we say they map keys to values. Each is called a key-value pair.

The key ‘unlocks’ the value. A key should be easy to remember and not change over time. The value can be more complicated and may change over time.

We will now express the same list as above using Python dictionary terminology.

• Python dictionary is a collection of objects (keys and values)
• Each key has an associated value
• A key can have many values
• As time changes, more values can be added to a key (values are mutable)
• A key cannot change (keys are immutable)
• Each key appears exactly once
• Keys can have the same value

Note: we can order dictionaries if we want but it is not necessary to do so. We’ll explain all these concepts in more detail throughout the article. But before we do anything, we need to know how to create a dictionary!

Python Create Dictionary

There are two ways to create a dictionary in Python:

1. Curly braces { }
2. The dict() constructor

Curly Braces { }

my_dict = {key1: value1,
           key2: value2,
           key3: value3,
           key4: value4,
           key5: value5}

We write the key, immediately followed by a colon. Then a single space, the value and finally a comma. After the last pair, replace the comma with a closing curly brace.

You can write all pairs on the same line. I put each on a separate line to aid readability.

Let’s say you have 5 friends and want to record which country they are from. You would write it like so (names and countries start with the same letter to make them easy to remember!).

names_and_countries = {'Adam': 'Argentina',
                       'Beth': 'Bulgaria',
                       'Charlie': 'Colombia',
                       'Dani': 'Denmark',
                       'Ethan': 'Estonia'}

The dict() Constructor

Option 1 – fastest to type

my_dict = dict(key1=value1,
               key2=value2,
               key3=value3,
               key4=value4,
               key5=value5)

So names_and_countries becomes

names_and_countries = dict(Adam='Argentina',
                           Beth='Bulgaria',
                           Charlie='Colombia',
                           Dani='Denmark',
                           Ethan='Estonia')

Each pair is like a keyword argument in a function. Keys are automatically converted to strings but values must be typed as strings.

Option 2 – slowest to type, best used with zip()

my_dict = dict([(key1, value1),
                (key2, value2),
                (key3, value3),
                (key4, value4),
                (key5, value5)])

names_and_countries becomes

names_and_countries = dict([('Adam', 'Argentina'),
                            ('Beth', 'Bulgaria'),
                            ('Charlie', 'Colombia'),
                            ('Dani', 'Denmark'),
                            ('Ethan', 'Estonia')])

Like with curly braces, we must explicitly type strings as strings. If you forget the quotes, Python interprets it as a function.

Option 2 with zip() – Python list to dict

If you have two lists and want to make a dictionary from them, do this

names = ['Adam', 'Beth', 'Charlie', 'Dani', 'Ethan']
countries = ['Argentina', 'Bulgaria', 'Colombia', 'Denmark', 'Estonia']
 
# Keys are names, values are countries
names_and_countries = dict(zip(names, countries))
 
>>> names_and_countries
{'Adam': 'Argentina',
'Beth': 'Bulgaria',
'Charlie': 'Colombia',
'Dani': 'Denmark',
'Ethan': 'Estonia'}

If you have more than two lists, do this

names = ['Adam', 'Beth', 'Charlie', 'Dani', 'Ethan']
countries = ['Argentina', 'Bulgaria', 'Colombia', 'Denmark', 'Estonia']
ages = [11, 24, 37, 75, 99]
 
# Zip all values together
values = zip(countries, ages)
 
# Keys are names, values are the tuple (countries, ages)
people_info = dict(zip(names, values))
 
>>> people_info
{'Adam': ('Argentina', 11),
'Beth': ('Bulgaria', 24),
'Charlie': ('Colombia', 37),
'Dani': ('Denmark', 75),
'Ethan': ('Estonia', 99)}

This is the first time we’ve seen a dictionary containing more than just strings! We’ll soon find out what can and cannot be a key or value. But first, let’s see how to access our data.

Accessing Key-Value Pairs

There are 2 ways to access the data in our dictionaries:

• Bracket notation [ ]
• The get() method

Bracket Notation [ ]

# Get value for the key 'Adam'
>>> names_and_countries['Adam']
'Argentina'
 
# Get value for the key 'Charlie'
>>> names_and_countries['Charlie']
'Colombia'
 
# KeyError if you search for a key not in the dictionary
>>> names_and_countries['Zoe']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'Zoe'

Type the key into the square brackets to get the corresponding value. If you enter a key not in the dictionary, Python raises a KeyError.

This looks like list indexing but it is completely different! For instance, you cannot access values by their relative position or by slicing.

# Not the first element of the dictionary
>>> names_and_countries[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 0
 
# Not the last element
>>> names_and_countries[-1]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: -1
 
# You cannot slice
>>> names_and_countries['Adam':'Dani']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'slice'

Python expects everything between the brackets to be a key. So for the first two examples, we have a KeyError because neither 0 nor -1 are keys in the dictionary. But is possible to use 0 or -1 as a key, as we will see soon.

Note: As of Python 3.7, the order elements are added is preserved. Yet you cannot use this order to access elements. It is more for iteration and visual purposes as we will see later.

If we try to slice our dictionary, Python raises a TypeError. We explain why in the Hashing section.

Let’s look at the second method for accessing the data stored in our dictionary.

Python Dictionary get() Method

# Get value for the key 'Adam'
>>> names_and_countries.get('Adam')
'Argentina'
 
# Returns None if key not in the dictionary
>>> names_and_countries.get('Zoe')
 
# Second argument returned if key not in dictionary
>>> names_and_countries.get('Zoe', 'Name not in dictionary')
'Name not in dictionary'
 
# Returns value if key in dictionary
>>> names_and_countries.get('Charlie', 'Name not in dictionary')
'Colombia'

The get() method takes two arguments:

1. The key you wish to search for
2. (optional) Value to return if the key is not in the dictionary (default is None).

It works like bracket notation. But it will never raise a KeyError. Instead, it returns either None or the object you input as the second argument.

This is hugely beneficial if you are iterating over a dictionary. If you use bracket notation and encounter an error, the whole iteration will stop. If you use get(), no error will be raised and the iteration will complete.

We will see how to iterate over dictionaries soon. But there is no point in doing that if we don’t even know what our dictionary can contain! Let’s learn about what can and can’t be a key-value pair.

Python Dict Keys

In real dictionaries, the spelling of words doesn’t change. It would make it quite difficult to use one if they did. The same applies to Python dictionaries. Keys cannot change. But they can be more than just strings. In fact, keys can be any immutable data type: string, int, float, bool or tuple.

>>> string_dict = {'hi': 'hello'}
>>> int_dict = {1: 'hello'}
>>> float_dict = {1.0: 'hello'}
>>> bool_dict = {True: 'hello', False: 'goodbye'}
>>> tuple_dict = {(1, 2): 'hello'}
 
# Tuples must only contain immutable types
>>> bad_tuple_dict = {(1, [2, 3]): 'hello'}
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unhashable type: 'list'

This is the second time we’ve seen "TypeError: unhashable type: 'list'". So what does ‘unhashable’ mean? 

What is Hashing in Python?

In the background, a Python dictionary is a data structure known as a hash table. It contains keys and hash values (numbers of fixed length). You apply hash() to a key to return its hash value. If we call hash() on the same key many times, the result will not change.

# Python 3.8 (different versions may give different results)
>>> hash('hello world!')
1357213595066920515
 
# Same result as above
>>> hash('hello world!')
1357213595066920515
 
# If we change the object, we change the hash value
>>> hash('goodbye world!')
-7802652278768862152

When we create a key-value pair, Python creates a hash-value pair in the background

# We write
>>> {'hello world!': 1}
 
# Python executes in the background
>>> {hash('hello world!'): 1}
 
# This is equivalent to
>>> {1357213595066920515: 1}

Python uses this hash value when we look up a key-value pair. By design, the hash function can only be applied to immutable data types. If keys could change, Python would have to create a new hash table from scratch every time you change them. This would cause huge inefficiencies and many bugs.

Instead, once a table is created, the hash value cannot change. Python knows which values are in the table and doesn’t need to calculate them again. This makes dictionary lookup and membership operations instantaneous and of O(1).

In Python, the concept of hashing only comes up when discussing dictionaries. Whereas, mutable vs immutable data types come up everywhere. Thus we say that you can only use immutable data types as keys, rather than saying ‘hashable’ data types.

Finally, what happens if you use the hash value of an object as another key in the same dictionary? Does Python get confused?

>>> does_this_work = {'hello': 1,
   			   hash('hello'): 2}
 
>>> does_this_work['hello']
1
 
>>> does_this_work[hash('hello')]
2

It works! The reasons why are beyond the scope of this article. The full implementation of the algorithm and the reasons why it works are described here. All you really need to know is that Python always picks the correct value… even if you try to confuse it! 

Python Dictionary Values

There are restrictions on dictionary keys but values have none. Literally anything can be a value. As long as your key is an immutable data type, your key-value pairs can be any combination of types you want. You have complete control!

>>> crazy_dict = {11.0: ('foo', 'bar'),
                  'baz': {1: 'a', 2: 'b'},
                  (42, 55): {10, 20, 30},
                  True: False}
 
# Value of the float 11.0 is a tuple
>>> crazy_dict[11.0]
('foo', 'bar')
 
# Value of the string 'baz' is a dictionary
>>> crazy_dict.get('baz')
{1: 'a', 2: 'b'}
 
# Value of the tuple (42, 55) is a set
>>> crazy_dict[(42, 55)]
{10, 20, 30}
 
# Value of the Bool True is the Bool False
>>> crazy_dict.get(True)
False

Note: you must use braces notation to type a dictionary out like this. If you try to use the dict() constructor, you will get SyntaxErrors (unless you use the verbose method and type out a list of tuples… but why would you do that?).

If you need to refresh your basic knowledge of Python sets, I recommend reading the ultimate guide to Python sets on the Finxter blog.

Python Nested Dictionaries

When web scraping, it is very common to work with dictionaries inside dictionaries (nested dictionaries). To access values on deeper levels, you simply chain methods together. Any order of bracket notation and get() is possible.

# Returns a dict
>>> crazy_dict.get('baz')
{1: 'a', 2: 'b'}
 
# Chain another method to access the values of this dict
>>> crazy_dict.get('baz').get(1)
'a'
 
>>> crazy_dict.get('baz')[2]
'b'

We now know how to create a dictionary and what data types are allowed where. But what if you’ve already created a dictionary and want to add more values to it? 

Python Add To Dictionary

>>> names_and_countries
{'Adam': 'Argentina', 
'Beth': 'Bulgaria', 
'Charlie': 'Colombia', 
'Dani': 'Denmark', 
'Ethan': 'Estonia'}
 
# Add key-value pair 'Zoe': 'Zimbabwe'
>>> names_and_countries['Zoe'] = 'Zimbabwe'
 
# Add key-value pair 'Fred': 'France'
>>> names_and_countries['Fred'] = 'France'
 
# Print updated dict
>>> names_and_countries
{'Adam': 'Argentina', 
'Beth': 'Bulgaria', 
'Charlie': 'Colombia', 
'Dani': 'Denmark', 
'Ethan': 'Estonia', 
'Zoe': 'Zimbabwe',     # Zoe first
'Fred': 'France'}      # Fred afterwards

Our dictionary reflects the order we added the pairs by first showing Zoe and then Fred.

To add a new key-value pair, we simply assume that key already exists and try to access it via bracket notation

>>> my_dict['new_key']

Then (before pressing return) use the assignment operator ‘=’  and provide a value.

>>> my_dict['new_key'] = 'new_value'

You cannot assign new key-value pairs via the get() method because it’s a function call.

>>> names_and_countries.get('Holly') = 'Hungary'
File "<stdin>", line 1
SyntaxError: cannot assign to function call

To delete a key-value pair use the del statement. To change the value of an existing key, use the same bracket notation as above.

# Delete the Zoe entry
>>> del names_and_countries['Zoe']
 
# Change Ethan's value
>>> names_and_countries['Ethan'] = 'DIFFERENT_COUNTRY'
 
>>> names_and_countries
{'Adam': 'Argentina', 
'Beth': 'Bulgaria', 
'Charlie': 'Colombia', 
'Dani': 'Denmark', 
'Ethan': 'DIFFERENT_COUNTRY',  # Ethan has changed
'Fred': 'France'}    		  # We no longer have Zoe

As with other mutable data types, be careful when using the del statement in a loop. It modifies the dictionary in place and can lead to unintended consequences. Best practice is to create a copy of the dictionary and to change the copy. Or you can use, my personal favorite, dictionary comprehensions (which we will cover later)—a powerful feature similar to the popular list comprehension feature in Python.

Python Dict Copy Method

>>> my_dict = {'a': 1, 'b': 2}
 
# Create a shallow copy
>>> shallow_copy = my_dict.copy()
 
# Create a deep copy
>>> import copy
>>> deep_copy = copy.deepcopy(my_dict)

To create a shallow copy of a dictionary use the copy() method. To create a deep copy use the deepcopy() method from the built-in copy module. We won’t discuss the distinction between the copy methods in this article for brevity.

Checking Dictionary Membership

Let’s say we have a dictionary with 100k key-value pairs. We cannot print it to the screen and visually check which key-value pairs it contains.

Thankfully, the following syntax is the same for dictionaries as it is for other objects such as lists and sets. We use the in keyword.

# Name obviously not in our dict
>>> 'INCORRECT_NAME' in names_and_countries
False
 
# We know this is in our dict
>>> 'Adam' in names_and_countries
True
 
# Adam's value is in the dict... right?
>>> names_and_countries['Adam']
'Argentina'
>>> 'Argentina' in names_and_countries
False

We expect INCORRECT_NAME not to be in our dict and Adam to be in it. But why does ‘Argentina’ return False? We’ve just seen that it’s the value of Adam?!

Remember at the start of the article that I said dictionaries are maps? They map from something we know (the key) to something we don’t (the value). So when we ask if something is in our dictionary, we are asking if it is a key. We’re not asking if it’s a value.

Which is more natural when thinking of a real-life dictionary:

1. Is the word ‘facetious’ in this dictionary?
2. Is the word meaning ‘lacking serious intent; concerned with something nonessential, amusing or frivolous’ in this dictionary?

Clearly the first one is the winner and this is the default behavior for Python.

>>> 'something' in my_dict

We are checking if ‘something’ is a key in my_dict.

But fear not, if you want to check whether a specific value is in a dictionary, that is possible! We simply have to use some methods. 

Python Dictionary Methods – Keys, Values and Items

There are 3 methods to look at. All can be used to check membership or for iterating over specific parts of a dictionary. Each returns an iterable.

• .keys() – iterate over the dictionary’s keys
• .values() – iterate over the dictionary’s values
• .items() – iterate over both the dictionary’s keys and values

Note: we’ve changed Ethan’s country back to Estonia for readability.

>>> names_and_countries.keys()
dict_keys(['Adam', 'Beth', 'Charlie', 'Dani', 'Ethan', 'Fred'])
 
>>> names_and_countries.values()
dict_values(['Argentina', 'Bulgaria', 'Colombia', 'Denmark', 'Estonia', 'France'])
 
>>> names_and_countries.items()
 
 
dict_items([('Adam', 'Argentina'), 
            ('Beth', 'Bulgaria'), 
            ('Charlie', 'Colombia'), 
            ('Dani', 'Denmark'), 
            ('Ethan', 'Estonia'), 
            ('Fred', 'France')])

We can now check membership in keys and values:

# Check membership in dict's keys
>>> 'Adam' in names_and_countries
True
>>> 'Adam' in names_and_countries.keys()
True
 
# Check membership in the dict's values
>>> 'Argentina' in names_and_countries.values()
True
 
# Check membership in either keys or values???
>>> 'Denmark' in names_and_countries.items()
False

You cannot check in the keys and values at the same time. This is because items() returns an iterable of tuples. As 'Denmark' is not a tuple, it will return False.

>>> for thing in names_and_countries.items():
  	  print(thing)
('Adam', 'Argentina')
('Beth', 'Bulgaria')
('Charlie', 'Colombia')
('Dani', 'Denmark')
('Ethan', 'Estonia')
 
# True because it's a tuple containing a key-value pair
>>> ('Dani', 'Denmark') in names_and_countries.items()
True

Python Loop Through Dictionary – An Overview

To iterate over any part of the dictionary we can use a for loop

>>> for name in names_and_countries.keys():
        print(name)
Adam
Beth
Charlie
Dani
Ethan
Fred
 
>>> for country in names_and_countries.values():
        print(f'{country} is wonderful!')
Argentina is wonderful!
Bulgaria is wonderful!
Colombia is wonderful!
Denmark is wonderful!
Estonia is wonderful!
France is wonderful!
 
>>> for name, country in names_and_countries.items():
        print(f'{name} is from {country}.')
Adam is from Argentina.
Beth is from Bulgaria.
Charlie is from Colombia.
Dani is from Denmark.
Ethan is from Estonia.
Fred is from France.

It’s best practice to use descriptive names for the objects you iterate over. Code is meant to be read and understood by humans! Thus we chose ‘name’ and ‘country’ rather than ‘key’ and ‘value’.

# Best practice
>>> for descriptive_key, descriptive_value in my_dict.items():
        # do something
 
# Bad practice (but you will see it 'in the wild'!)
>>> for key, value in my_dict.items():
        # do something

If your key-value pairs don’t follow a specific pattern, it’s ok to use ‘key’ and ‘value’ as your iterable variables, or even ‘k’ and ‘v’.

# Iterating over the dict is the same as dict.keys()
>>> for thing in names_and_countries:
        print(thing)
Adam
Beth
Charlie
Dani
Ethan
Fred

A Note On Reusability

# Works with general Python types
>>> for key in object:
        # do something
 
# Works only with dictionaries
>>> for key in object.keys():
        # do something

Do not specify keys() if your code needs to work with other objects like lists and sets. Use the keys() method if your code is only meant for dictionaries. This prevents future users inputting incorrect objects.

Python dict has_key

The method has_key() is exclusive to Python 2. It returns True if the key is in the dictionary and False if not.

Python 3 removed this functionality in favour of the following syntax:

>>> if key in d:
        # do something

This keeps dictionary syntax in line with that of other data types such as sets and lists. This aids readability and reusability.

Pretty Printing Dictionaries Using pprint()

The built-in module pprint contains the function pprint. This will ‘pretty print’ your dictionary. It sorts the keys alphabetically and prints each key-value pair on a newline.

>>> from pprint import pprint
>>> messy_dict = dict(z='Here is a really long key that spans a lot of text', a='here is another long key that is really too long', j='this is the final key in this dictionary')
 
>>> pprint(messy_dict)
{'a': 'here is another long key that is really too long',
'j': 'this is the final key in this dictionary',
'z': 'Here is a really long key that spans a lot of text'}

It does not change the dictionary at all. It’s just much more readable now.

Python Dictionaries and JSON Files

We need to encode and decode all this data.

A common filetype you will interact with is a JSON file. It stands for Javascript Object Notation. They are used to structure and send data in web applications.

They work almost exactly the same way as dictionaries and you can easily turn one into the other very easily.

Python Dict to JSON

>>> import json
>>> my_dict = dict(a=1, b=2, c=3, d=4)
 
>>> with open('my_json.json', 'w') as f:
   	 json.dump(my_dict, f)

The above code takes my_dict and writes it to the file my_json.json in the current directory.

You can get more complex than this by setting character encodings and spaces. For more detail, we direct the reader to the docs.

Python JSON to Dict

We have the file my_json.json in our current working directory.

>>> import json
>>> with open('my_json.json', 'r') as f:
        new_dict = json.load(f)
 
>>> new_dict
{'a': 1, 'b': 2, 'c': 3, 'd': 4}

Note: the key-value pairs in JSON are always converted to strings when encoded in Python. It is easy to change any object into a string and it leads to fewer errors when encoding and decoding files. But it means that sometimes the file you load and the file you started with are not identical. 

Python Dictionary Methods

Here’s a quick overview:

1. dict.clear() – remove all key-value pairs from a dict
2. dict.update() – merge two dictionaries together
3. dict.pop() – remove a key and return its value
4. dict.popitem() – remove a random key-value pair and return it as a tuple

We’ll use letters A and B for our dictionaries as they are easier to read than descriptive names. Plus we have kept the examples simple to aid understanding.

dict.clear() – remove all key-value pairs from a dict

>>> A = dict(a=1, b=2)
>>> A.clear()
>>> A
{}

Calling this on a dict removes all key-value pairs in place. The dict is now empty.

dict.update() – merge two dictionaries together

>>> A = dict(a=1, b=2)
>>> B = dict(c=3, d=4)
>>> A.update(B)
>>> A
{'a': 1, 'b': 2, 'c': 3, 'd': 4}
>>> B
{'c': 3, 'd': 4}

We have just updated A. Thus all the key-value pairs from B have been added to A. B has not changed.

If A and B some keys, B’s value will replace A’s. This is because A is updated by B and so takes all of B’s values (not the other way around).

>>> A = dict(a=1, b=2)
>>> B = dict(b=100)
>>> A.update(B)
 
# A now contains B's values
>>> A
{'a': 1, 'b': 100}
 
# B is unchanged
>>> B
{'b': 100}

You can also pass a sequence of tuples or keyword arguments to update(), like you would with the dict() constructor.

>>> A = dict(a=1, b=2)
# Sequence of tuples
>>> B = [('c', 3), ('d', 4)]
>>> A.update(B)
>>> A
{'a': 1, 'b': 2, 'c': 3, 'd': 4}
 
>>> A = dict(a=1, b=2)
# Pass key-value pairs as keyword arguments
>>> A.update(c=3, d=4)
>>> A
{'a': 1, 'b': 2, 'c': 3, 'd': 4}

dict.pop() – remove a key and return its value

>>> A = dict(a=1, b=2)
>>> A.pop('a')
1
>>> A
{'b': 2}

If you try call dict.pop() with a key that is not in the dictionary, Python raises a KeyError.

>>> A.pop('non_existent_key')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'non_existent_key'

Like the get() method, you can specify an optional second argument. This is returned if the key is not in the dictionary and so avoids KeyErrors.

>>> A.pop('non_existent_key', 'not here')
'not here'

dict.popitem() – remove a random key-value pair and return it as a tuple

>>> A = dict(a=1, b=2, c=3)
# Your results will probably differ
>>> A.popitem()
('c', 3)
>>> A
{'a': 1, 'b': 2}
>>> A.popitem()
('b', 2)
>>> A
{'a': 1}

If the dictionary is empty, Python raises a KeyError. 

>>> A = dict()
>>> A.popitem()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'popitem(): dictionary is empty'

Python Loop Through Dictionary – In Detail

There are several common situations you will encounter when iterating over dictionaries. Python has developed several methods to help you work more efficiently.

But before we head any further, please remember the following:

NEVER EVER use bracket notation when iterating over a dictionary. If there are any errors, the whole iteration will break and you will not be happy.

The standard Python notation for incrementing numbers or appending to lists is

# Counting
my_num = 0
for thing in other_thing:
    my_num += 1
 
# Appending to lists
my_list = []
for thing in other_thing:
    my_list.append(thing)

This follows the standard pattern:

1. Initialise ’empty’ object
2. Begin for loop
3. Add things to that object

When iterating over a dictionary, our values can be numbers or list-like. Thus we can add or we can append to values. It would be great if our code followed the above pattern. But…

>>> my_dict = {}
>>> for thing in other_thing:
        my_dict['numerical_key'] += 1
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
KeyError: 'numerical_key'
 
>>> for thing in other_thing:
        my_dict['list_key'].append(thing)
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
KeyError: 'list_key'

Unfortunately, both raise a KeyError. Python tells us the key do not exist and so we cannot increment its value. Thus we must first create a key-value pair before we do anything with it.

We’ll now show 4 ways to solve this problem:

1. Manually initialise a key if it does not exist
2. The get() method
3. The setdefault() method
4. The defaultdict()

We’ll explain this through some examples, so let’s go to the setup.

Three friends – Adam, Bella and Cara, have gone out for a meal on Adam’s birthday. They have stored their starter, main and drinks orders in one list. The price of each item is in another list. We will use this data to construct different dictionaries.

people = ['Adam', 'Bella', 'Cara',
          'Adam', 'Bella', 'Cara',
          'Adam', 'Bella', 'Cara',]
 
food = ['soup', 'bruschetta', 'calamari',   # starter
        'burger', 'calzone', 'pizza',       # main
        'coca-cola', 'fanta', 'water']      # drink
 
# Cost of each item in £
prices = [3.20, 4.50, 3.89,
          12.50, 15.00, 13.15,
          3.10, 2.95, 1.86]
 
# Zip data together to allow iteration
# We only need info about the person and the price
meal_data = zip(people, prices)

Our three friends are very strict with their money. They want to pay exactly the amount they ordered. So we will create a dictionary containing the total cost for each person. This is a numerical incrementation problem.

Manually Initialize a Key

# Initialise empty dict
total = {}
 
# Iterate using descriptive object names
for (person, price) in meal_data:
 
    # Create new key and set value to 0 if key doesn't yet exist
    if person not in total:
        total[person] = 0
    
    # Increment the value by the price of each item purchased.
    total[person] += price
 
>>> total
{'Adam': 18.8, 'Bella': 22.45, 'Cara': 18.9}

We write an if statement which checks if the key is already in the dictionary. If it isn’t, we set the value to 0. If it is, Python does not execute the if statement. We then increment using the expected syntax.

This works well but requires quite a few lines of code. Surely we can do better?

Python Dict get() Method When Iterating

# Reinitialise meal_data as we have already iterated over it
meal_data = zip(people, prices)
 
total = {}
for (person, price) in meal_data:
 
    # get method returns 0 the first time we call it
    # and returns the current value subsequent times
    total[person] = total.get(person, 0) + price
 
>>> total
{'Adam': 18.8, 'Bella': 22.45, 'Cara': 18.9}

We’ve got it down to one line!

We pass get() a second value which is returned if the key is not in the dictionary. In this case, we choose 0 like the above example. The first time we call get() it returns 0. We have just initialised a key-value pair! In the same line, we add on ‘price’. The next time we call get(), it returns the current value and we can add on ‘price’ again.

This method does not work for appending. You need some extra lines of code. We will look at the setdefault() method instead.

Python Dict setdefault() Method

The syntax of this method makes it an excellent choice for modifying a key’s value via the append() method.

First we will show why it’s not a great choice to use if you are incrementing with numbers.

meal_data = zip(people, prices)
total = {}
for (person, price) in meal_data:
 
    # Set the initial value of person to 0
    total.setdefault(person, 0)
 
    # Increment by price
    total[person] += price
 
0
0
0
3.2
4.5
3.89
15.7
19.5
17.04
>>> total
{'Adam': 18.8, 'Bella': 22.45, 'Cara': 18.9}

It works but requires more lines of code than get() and prints lots of numbers to the screen. Why is this?

The setdefault() method takes two arguments:

1. The key you wish to set a default value for
2. What you want the default value to be

So setdefault(person, 0) sets the default value of person to be 0.

It always returns one of two things:

1. The current value of the key
2. If the key does not exist, it returns the default value provided

This is why the numbers are printed to the screen. They are the values of ‘person’ at each iteration.

Clearly this is not the most convenient method for our current problem. If we do 100k iterations, we don’t want 100k numbers printed to the screen.

So we recommend using the get() method for numerical calculations.

Let’s see it in action with lists and sets. In this dictionary, each person’s name is a key. Each value is a list containing the price of each item they ordered (starter, main, dessert). 

meal_data = zip(people, prices)
individual_bill = {}
 
for (person, price) in meal_data:
 
    # Set default to empty list and append in one line!
    individual_bill.setdefault(person, []).append(price)
 
>>> individual_bill
{'Adam': [3.2, 12.5, 3.1], 
'Bella': [4.5, 15.0, 2.95], 
'Cara': [3.89, 13.15, 1.86]}

Now we see the true power of setdefault()! Like the get method in our numerical example, we initialise a default value and modify it in one line!

Note: setdefault() calculates the default value every time it is called. This may be an issue if your default value is expensive to compute. Get() only calculates the default value if the key does not exist. Thus get() is a better choice if your default value is expensive. Since most default values are ‘zeros’ such as 0, [ ] and { }, this is not an issue for most cases.

We’ve seen three solutions to the problem now. We’ve got the code down to 1 line. But the syntax for each has been different to what we want. Now let’s see something that solves the problem exactly as we’d expect: introducing defaultdict!

Python defaultdict()

Let’s solve our numerical incrementation problem:

# Import from collections module
from collections import defaultdict
 
meal_data = zip(people, prices)
 
# Initialise with int to do numerical incrementation
total = defaultdict(int)
 
# Increment exactly as we want to!
for (person, price) in meal_data:
    total[person] += price
 
>>> total
defaultdict(<class 'int'>, {'Adam': 18.8, 'Bella': 22.45, 'Cara': 18.9})

Success!! But what about our list problem?

from collections import defaultdict
 
meal_data = zip(people, prices)
 
# Initialise with list to let us append
individual_bill = defaultdict(list)
 
for (person, price) in meal_data:
    individual_bill[person].append(price)
 
>>> individual_bill
defaultdict(<class 'list'>, {'Adam': [3.2, 12.5, 3.1], 
                             'Bella': [4.5, 15.0, 2.95], 
                             'Cara': [3.89, 13.15, 1.86]})

The defaultdict is part of the built-in collections module. So before we use it, we must first import it.

Defaultdict is the same as a normal Python dictionary except:

1. It takes a callable data type as an argument
2. When it meets a key for the first time, the default value is set as the ‘zero’ for that data type. For int it is 0, for list it’s an empty list [ ] etc..

Thus you will never get a KeyError! Plus and initialising default values is taken care of automatically!

We have now solved the problem using the same syntax for lists and numbers!

Now let’s go over some special cases for defaultdict.

Python defaultdict() Special Cases

Above we said it’s not possible to get a KeyError when using defaultdict. This is only true if you correctly initialise your dict.

# Initialise without an argument
>>> bad_dict = defaultdict()
>>> bad_dict['key']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'key'
 
# Initialise with None
>>> another_bad_dict = defaultdict(None)
>>> another_bad_dict['another_key']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: 'another_key'

Let’s say you initialise defaultdict without any arguments. Then Python raises a KeyError if you call a key not in the dictionary. This is the same as initialising with None and defeats the whole purpose of defaultdict.

The issue is that None is not callable. Yet you can get defaultdict to return None by using a lambda function:

>>> none_dict = defaultdict(lambda: None)
>>> none_dict['key']
>>>

Note that you cannot increment or append to None. Make sure you choose your default value to match the problem you are solving!

Whilst we’re here, let’s take a look at some more dictionaries in the collections module.

OrderedDict 

Earlier we said that dictionaries preserve their order from Python 3.7 onwards. So why do we need something called OrderedDict?

As the name suggests, OrderedDict preserves the order elements are added. But two OrderedDicts are the same if and only if their elements are in the same order. This is not the case with normal dicts.

>>> from collections import OrderedDict
 
# Normal dicts preserve order but don't use it for comparison
>>> normal1 = dict(a=1, b=2)
>>> normal2 = dict(b=2, a=1)
>>> normal1 == normal2
True
 
# OrderedDicts preserve order and use it for comparison
>>> ordered1 = OrderedDict(a=1, b=2)
>>> ordered2 = OrderedDict(b=2, a=1)
>>> ordered1 == ordered2
False

Other than that, OrderedDict has all the same properties as a regular dictionary. If your elements must be in a particular order, then use OrderedDict!

Counter()

Let’s say we want to count how many times each word appears in a piece of text (a common thing to do in NLP). We’ll use The Zen of Python for our example. If you don’t know what it is, run

>>> import this

I’ve stored it in the list zen_words where each element is a single word.

We can manually count each word using defaultdict. But printing it out with the most frequent words occurring first is a bit tricky.

>>> from collections import defaultdict
>>> word_count = defaultdict(int)
>>> for word in zen_words:
        word_count[word] += 1
 
# Define function to return the second value of a tuple
>>> def select_second(tup):
        return tup[1]
 
# Reverse=True - we want the most common first
# word_count.items() - we want keys and values
# sorted() returns a list, so wrap in dict() to return a dict
 
>>> dict(sorted(word_count.items(), reverse=True, key=select_second))
{'is': 10, 
'better': 8, 
'than': 8, 
'to': 5, 
...}

As counting is quite a common process, the Counter() dict subclass was created. It is complex enough that we could write a whole article about it. 

For brevity, we will include the most basic use cases and let the reader peruse the docs themselves.

>>> from collections import Counter
>>> word_count = Counter(zen_words)
>>> word_count
Counter({'is': 10, 'better': 8, 'than': 8, 'to': 5, ...})

You can pass any iterable or dictionary to Counter(). It returns a dictionary in descending order of counts

>>> letters = Counter(['a', 'b', 'c', 'c', 'c', 'c', 'd', 'd', 'a'])
>>> letters
Counter({'c': 4, 'a': 2, 'd': 2, 'b': 1})
 
# Count of a missing key is 0
>>> letters['z']
0

Reversed()

In Python 3.8 they introduced the reversed() function for dictionaries! It returns an iterator. It iterates over the dictionary in the opposite order to how the key-value pairs were added. If the key-value pairs have no order, reversed() will not give them any further ordering. If you want to sort the keys alphabetically for example, use sorted().

# Python 3.8
 
# Reverses the order key-value pairs were added to the dict
>>> ordered_dict = dict(a=1, b=2, c=3)
>>> for key, value in reversed(ordered_dict.items()):
        print(key, value)
c 3
b 2
a 1
 
# Does not insert order where there is none.
>>> unordered_dict = dict(c=3, a=1, b=2)
>>> for key, value in reversed(unordered_dict.items()):
        print(key, value)
b 2
a 1
c 3
 
# Order unordered_dict alphabetically using sorted()
>>> dict(sorted(unordered_dict.items()))
{'a': 1, 'b': 2, 'c': 3}

Since it’s an iterator, remember to use the keys(), values() and items() methods to select the elements you want. If you don’t specify anything, you’ll iterate over the keys.

Dictionary Comprehensions

A wonderful feature of dictionaries, and Python in general, is the comprehension. This lets you create dictionaries in a clean, easy to understand and Pythonic manner. You must use curly braces {} to do so (not dict()).

We’ve already seen that if you have two lists, you can create a dictionary from them using dict(zip()). 

names = ['Adam', 'Beth', 'Charlie', 'Dani', 'Ethan']
countries = ['Argentina', 'Bulgaria', 'Colombia', 'Denmark', 'Estonia']
 
dict_zip = dict(zip(names, countries))
 
>>> dict_zip
{'Adam': 'Argentina',
'Beth': 'Bulgaria',
'Charlie': 'Colombia',
'Dani': 'Denmark',
'Ethan': 'Estonia'}

We can also do this using a for loop

>>> new_dict = {}
>>> for name, country in zip(names, countries):
        new_dict[name] = country
 
>>> new_dict
{'Adam': 'Argentina',
'Beth': 'Bulgaria',
'Charlie': 'Colombia',
'Dani': 'Denmark',
'Ethan': 'Estonia'}

We initialize our dict and iterator variables with descriptive names. To iterate over both lists at the same time we zip them together. Finally, we add key-value pairs as desired. This takes 3 lines.

Using a comprehension turns this into one line.

dict_comp = {name: country for name, country in zip(names, countries)}
 
>>> dict_comp
{'Adam': 'Argentina',
'Beth': 'Bulgaria',
'Charlie': 'Colombia',
'Dani': 'Denmark',
'Ethan': 'Estonia'}

They are a bit like for loops in reverse. First, we state what we want our key-value pairs to be. Then we use the same for loop as we did above. Finally, we wrap everything in curly braces.

Note that every comprehension can be written as a for loop. If you ever get results you don’t expect, try it as a for loop to see what is happening.

Here’s a common mistake

dict_comp_bad = {name: country 
                 for name in names 
                 for country in countries}
 
>>> dict_comp_bad
{'Adam': 'Estonia',
'Beth': 'Estonia',
'Charlie': 'Estonia',
'Dani': 'Estonia',
'Ethan': 'Estonia'}

What’s going on? Let’s write it as a for loop to see. First, we’ll write it out to make sure we are getting the same, undesired, result.

bad_dict = {}
for name in names:
    for country in countries:
        bad_dict[name] = country
 
>>> bad_dict
{'Adam': 'Estonia',
'Beth': 'Estonia',
'Charlie': 'Estonia',
'Dani': 'Estonia',
'Ethan': 'Estonia'}

Now we’ll use the bug-finder’s best friend: the print statement!

# Don't initialise dict to just check for loop logic
for name in names:
    for country in countries:
        print(name, country)
Adam Argentina
Adam Bulgaria
Adam Colombia
Adam Denmark
Adam Estonia
Beth Argentina
Beth Bulgaria
Beth Colombia
...
Ethan Colombia
Ethan Denmark
Ethan Estonia

Here we remove the dictionary to check what is actually happening in the loop. Now we see the problem! The issue is we have nested for loops. The loop says: for each name pair it with every country. Since dictionary keys can only appear, the value gets overwritten on each iteration. So each key’s value is the final one that appears in the loop – ‘Estonia’.

The solution is to remove the nested for loops and use zip() instead.

Python Nested Dictionaries with Dictionary Comprehensions

nums = [0, 1, 2, 3, 4, 5]
 
dict_nums = {n: {'even': n % 2 == 0,
                 'square': n**2,
                 'cube': n**3,
                 'square_root': n**0.5}
             for n in nums}
 
# Pretty print for ease of reading
>>> pprint(dict_nums)
{0: {'cube': 0, 'even': True, 'square': 0, 'square_root': 0.0},
1: {'cube': 1, 'even': False, 'square': 1, 'square_root': 1.0},
2: {'cube': 8, 'even': True, 'square': 4, 'square_root': 1.4142135623730951},
3: {'cube': 27, 'even': False, 'square': 9, 'square_root': 1.7320508075688772},
4: {'cube': 64, 'even': True, 'square': 16, 'square_root': 2.0},
5: {'cube': 125, 'even': False, 'square': 25, 'square_root': 2.23606797749979}}

This is where comprehensions become powerful. We define a dictionary within a dictionary to create lots of information in a few lines of code. The syntax is exactly the same as above but our value is more complex than the first example.

Remember that our key value pairs must be unique and so we cannot create a dictionary like the following

>>> nums = [0, 1, 2, 3, 4, 5]
>>> wrong_dict = {'number': num, 'square': num ** 2 for num in nums}
  File "<stdin>", line 1
    wrong_dict = {'number': num, 'square': num ** 2 for num in nums}
                                                    ^
SyntaxError: invalid syntax

We can only define one pattern for key-value pairs in a comprehension. But if you could define more, it wouldn’t be very helpful. We would overwrite our key-value pairs on each iteration as keys must be unique.

If-Elif-Else Statements

nums = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
 
# Just the even numbers
even_squares = {n: n ** 2 for n in nums
                if n % 2 == 0}
 
# Just the odd numbers
odd_squares = {n: n ** 2 for n in nums
               if n % 2 == 1}
 
>>> even_dict
{0: 0, 2: 4, 4: 16, 6: 36, 8: 64, 10: 100}
 
>>> odd_dict
{1: 1, 3: 9, 5: 25, 7: 49, 9: 81}

We can apply if conditions after the for statement. This affects all the values you are iterating over.

You can also apply them to your key and value definitions. We’ll now create different key-value pairs based on whether a number is odd or even.

# Use parenthesis to aid readability
different_vals = {n: ('even' if n % 2 == 0 else 'odd')
                  for n in range(5)}
 
>>> different_vals
{0: 'even', 1: 'odd', 2: 'even', 3: 'odd', 4: 'even'}

We can get really complex and use if/else statements in both the key-value definitions and after the for loop!

# Change each key using an f-string
{(f'{n}_cubed' if n % 2 == 1 else f'{n}_squared'): 
 
# Cube odd numbers, square even numbers
 (n ** 3 if n % 2 == 1 else n ** 2)
 
# The numbers 0-10 inclusive
 for n in range(11)
 
# If they are not multiples of 3
 if n % 3 != 0}
 
{'1_cubed': 1, '2_squared': 4, '4_squared': 16, '5_cubed': 125, '7_cubed': 343, '8_squared': 64, '10_squared': 100}

It is relatively simple to do this using comprehensions. Trying to do so with a for loop or dict() constructor would be much harder.

Merging Two Dictionaries

Let’s say we have two dictionaries A and B. We want to create a dictionary, C, that contains all the key-value pairs of A and B. How do we do this?

>>> A = dict(a=1, b=2)
>>> B = dict(c=3, d=4)
 
# Update method does not create a new dictionary
>>> C = A.update(B)
>>> C
>>> type(C)
<class 'NoneType'>
 
>>> A
{'a': 1, 'b': 2, 'c': 3, 'd': 4}

Using merge doesn’t work. It modifies A in place and so doesn’t return anything.

Before Python 3.5, you had to write a function to do this. In Python 3.5 they introduced this wonderful bit of syntax.

# Python >= 3.5
>>> A = dict(a=1, b=2)
>>> B = dict(c=3, d=4)
>>> C = {**A, **B}
>>> C
{'a': 1, 'b': 2, 'c': 3, 'd': 4}

We use ** before each dictionary to ‘unpack’ all the key-value pairs.

The syntax is very simple: a comma-separated list of dictionaries wrapped in curly braces. You can do this for an arbitrary number of dictionaries.

A = dict(a=1, b=2)
B = dict(c=3, d=4)
C = dict(e=5, f=6)
D = dict(g=7, h=8)
>>> all_the_dicts = {**A, **B, **C, **D}
>>> all_the_dicts
{'a': 1, 'b': 2, 'c': 3, 'd': 4, 'e': 5, 'f': 6, 'g': 7, 'h': 8}

Finally, what happens if the dicts share key-value pairs?

>>> A = dict(a=1, b=2)
>>> B = dict(a=999)
>>> B_second = {**A, **B}
>>> A_second = {**B, **A}
 
# Value of 'a' taken from B
>>> B_second
{'a': 999, 'b': 2}
 
# Value of 'a' taken from A
>>> A_second
{'a': 1, 'b': 2}

As is always the case with Python dictionaries, a key’s value is dictated by its last assignment. The dict B_second first takes A’s values then take’s B’s. Thus any shared keys between A and B will be overwritten with B’s values. The opposite is true for A_second.

Note: if a key’s value is overridden, the position of that key in the dict does not change.

>>> D = dict(g=7, h=8)
>>> A = dict(a=1, g=999)
>>> {**D, **A}
 
# 'g' is still in the first position despite being overridden with A's value
{'g': 999, 'h': 8, 'a': 1}

Conclusion

You now know almost everything you’ll ever need to know to use Python Dictionaries. Well done! Please bookmark this page and refer to it as often as you need!

If you have any questions post them in the comments and we’ll get back to you as quickly as possible.

If you love Python and want to become a freelancer, there is no better course out there than this one:

I bought it myself and it is why you are reading these words today. 

About the Author

This article is contributed by Finxter user Adam Murphy (data scientist, grandmaster of Python code):

I am a self-taught programmer with a First Class degree in Mathematics from Durham University and have been coding since June 2019.

I am well versed in the fundamentals of web scraping and data science and can get you a wide variety of information from the web very quickly.

I recently scraped information about all watches that Breitling and Rolex sell in just 48 hours and am confident I can deliver datasets of similar quality to you whatever your needs.

Being a native English speaker, my communication skills are excellent and I am available to answer any questions you have and will provide regular updates on the progress of my work.

If you want to hire Adam, check out his Upwork profile!

References

1. https://www.dictionary.com/
2. https://tinyurl.com/yg6kgy9h
3. https://stackoverflow.com/questions/7886355/defaultdictnone
4. https://www.datacamp.com/community/tutorials/python-dictionary-tutorial
5. https://docs.python.org/3.8/tutorial/datastructures.html#dictionaries
6. https://stackoverflow.com/questions/526125/why-is-python-ordering-my-dictionary-like-so
7. https://stackoverflow.com/a/378987/11829398
8. https://en.wikipedia.org/wiki/Hash_function
9. https://docs.python.org/2/library/collections.html#collections.OrderedDict
10. https://www.quora.com/What-are-hashable-types-in-Python
11. https://hg.python.org/cpython/file/default/Objects/dictobject.c
12. https://www.dictionary.com/browse/facetious?s=t
13. https://thispointer.com/python-how-to-copy-a-dictionary-shallow-copy-vs-deep-copy/
14. https://docs.python.org/3.8/library/collections.html#collections.Counter
15. https://stackoverflow.com/questions/12309269/how-do-i-write-json-data-to-a-file
16. https://realpython.com/python-dicts/#built-in-dictionary-methods
17. https://stackoverflow.com/questions/38987/how-do-i-merge-two-dictionaries-in-a-single-expression

Where to Go From Here?

Congratulations! You’ve successfully mastered dictionaries in Python! Now, let’s dive into Python sets.

The post Python Dictionary – The Ultimate Guide appeared first on Be on the Right Side of Change.

Try It Yourself:

Exercise: Modify each function and play with the output!

Here’s the code for copy&paste:

# 1. map(func, iter)
m = map(lambda x: x[0], ['red', 'green', 'blue'])
print(list(m))
# ['r', 'g', 'b']


# 2. map(func, i1, ...,ik)
l1 = [0, 2, 2]
l2 = ['apple', 'orange', 'banana']
m = map(lambda x, y: str(x) + ' ' + str(y) + 's', l1, l2)
print(list(m))
# ['0 apples', '2 oranges', '2 bananas']


# 3. string.join(iter)
print(' marries '.join(['Alice', 'Bob']))
# Alice marries Bob


# 4. filter(func, iterable)
print(list(filter(lambda x: x>17, [1, 15, 17, 18])))
# [18]


# 5. string.strip()
print("    \n   \t  42  \t ".strip())
# 42


# 6. sorted(iter)
print(sorted([8, 3, 2, 42, 5]))
# [2, 3, 5, 8, 42]


# 7. sorted(iter, key=key)
print(sorted([8, 3, 2, 42, 5], key=lambda x: 0 if x==42 else x))
# [42, 2, 3, 5, 8]


# 8. help(func)
help(str.upper)
'''
Help on method_descriptor:

upper(self, /)
    Return a copy of the string converted to uppercase.
'''


# 9. zip(i1, i2, ...)
print(list(zip(['Alice', 'Anna'], ['Bob', 'Jon', 'Frank'])))
# [('Alice', 'Bob'), ('Anna', 'Jon')]


# 10. Unzip
print(list(zip(*[('Alice', 'Bob'), ('Anna', 'Jon')])))
# [('Alice', 'Anna'), ('Bob', 'Jon')]


# 11. Enumerate
print(list(enumerate(['Alice', 'Bob', 'Jon'])))
# [(0, 'Alice'), (1, 'Bob'), (2, 'Jon')]

The following tutorials address those functions in great detail (including videos on the most important functions).

Related Articles:

• Map Function
• String Join
• Filtering Lists
• Strip and Trim
• Python Sorting
• Zip and Unzip

Know someone who would benefit from this cheat sheet? Share it with your friends!

Simplify learning Python and register for the free 5x Python cheat sheet course.

The post Python Functions and Tricks Cheat Sheet appeared first on Be on the Right Side of Change.

Slicing is a Python-specific concept for carving out a range of values from sequence types such as lists or strings.

Try It Yourself:

Slicing is one of the most popular Python features. To master Python, you must master slicing first. Any non-trivial Python code base relies on slicing. In other words, the time you invest now in mastering slicing will be repaid a hundredfold during your career.

Download: Free Coffee Break Python Slicing eBook

Here’s a 10-min video version of this article–that shows you everything you need to know about slicing:

[Intermezzo] Indexing Basics

To bring everybody on the same page, let me quickly explain indices in Python by example. Suppose we have a string 'universe'. The indices are simply the positions of the characters of this string.

The first character has index 0, the second character has index 1, and the i-th character has index i-1.

Quick Introduction Slicing

Info: The idea of slicing is simple. You carve out a subsequence from a sequence by defining the start and end indices. But while indexing retrieves only a single character, slicing retrieves a whole substring within an index range.

Use the bracket notation for slicing with the start and end position identifiers. For example, word[i:j] returns the substring starting from index i (included) and ending in index j (excluded). Forgetting that the end index is excluded is a common source of bugs.

You can also skip the position identifier before or after the slicing colon. This indicates that the slice starts from the first or last position, respectively. For example, word[:i] + word[i:] returns the same string as word.

Python puzzle 1: What is the output of this code snippet?

x = 'universe'
print(x[2:4])

The Step Size in Slicing

For the sake of completeness, let me quickly explain the advanced slicing notation [start:end:step]. The only difference to the previous notation is that it allows you to specify the step size as well.

For example, the command 'python'[:5:2] returns every second character up to the fourth character, i.e., the string 'pto'.

Python puzzle 2: What is the output of this code snippet?

x = 'universe'
print(x[2::2])

Overshooting Indices in Slicing

Slicing is robust even if the end index shoots over the maximal sequence index. You only need to know that nothing unexpected happens if slicing overshoots sequence indices: it just slices to the maximum slice element.

Here is an example.

word = "galaxy" 
print(word[4:50]) 

Summary Python Slicing

Short recap, the slice notation s[start:end:step] carves out a substring from s. The substring consists of all characters between the two characters at the start index (inclusive) and the end index (exclusive). An optional step size indicates how many characters are left out from the original sequence. Here is an example:

s = 'sunshine'
print(s[1:5:2])
#'us'
print(s[1:5:1])
#'unsh'

Learn Slicing By Doing

Ok, so let’s train slicing a little bit. Solve the following puzzle in your head (and check the solution below).

Python Puzzle 3: What is the output of this code snippet?

# (Shakespeare)
s = "All that glitters is not gold"
print(s[9:-9])
print(s[::10])
print(s[:-4:-1])

Frequently Asked Questions

Let’s go a bit deeper into slicing to ensure you are getting it 100%.

I have searched the web to find all the little problems new Python coders face with slicing. I will answer six common questions next.

1) How to skip slicing indices (e.g., s[::2])?

The Python interpreter assumes certain default values for s[start:stop:step]. They are: start=0, stop=len(s), and step=1 (in the slice notation: s[::]==s[0:len(s):1]).

2) When to use the single colon notation (e.g., s[:]) and when double colon notation (e.g., s[::2])?

A single colon (e.g. s[1:2]) allows for two arguments, the start and the end index. A double colon (e.g. s[1:2:2]) allows for three arguments, the start index, the end index, and the step size. If the step size is set to the default value 1, we can use the single colon notation for brevity.

3) What does a negative step size mean (e.g., s[5:1:-1])?

This is an interesting feature in Python. A negative step size indicates that we are not slicing from left to right but from right to left. Hence, the start index should be larger or equal to the end index (otherwise, the resulting sequence is empty).

4) What are the default indices when using a negative step size (e.g., s[::-1])?

In this case, the default indices are not start=0 and end=len(s) but the other way round: start=len(s)-1 and end=-1. Note that the start index is still included, and the end index is still excluded from the slice. Because of that, the default end index is -1 and not 0.

5) We have seen many examples of string slicing. How does list slicing work?

Slicing works the same for all sequence types. For lists, consider the following example:

l = [1, 2, 3, 4]
print(l[2:])
# [3, 4]

Slicing tuples works in a similar way.

6) Why is the last index excluded from the slice?

The last index is excluded because of two reasons. The first reason is language consistency, e.g., the range function also does not include the end index. The second reason is clarity. Here’s an example of why it makes sense to exclude the end index from the slice.

customer_name = 'Hubert'
k = 3 # maximal size of database entry
x = 1 # offset
db_name = customer_name[x:x+k]

Now suppose the end index would be included. In this case, the total length of the db_name substring would be k + 1 characters. This would be very counter-intuitive.

Puzzle Solutions

Here are the solutions to the puzzles in this article.

Puzzle 1:

iv

Puzzle 2:

 ies 

Puzzle 3:

glitters is
 Al 
 dlo

Subscribe to *FREE* Python Training Newsletter

And become a Python master on auto-pilot:

You can also download our free slicing ebook here:

Download: Free Coffee Break Python Slicing eBook

The post Introduction to Slicing in Python appeared first on Be on the Right Side of Change.

1. Overcome the Knowledge Gap

The great teacher Socrates delivered complex knowledge by asking a sequence of questions. Each question built on answers to previous questions provided by the student. This more than 2400 year old teaching technique is still in widespread use today. A good teacher opens a gap between their knowledge and the learner’s. This knowledge gap makes the learner aware that they do not know the answer to a burning question. This creates a tension in the learner’s mind. To close this gap, the learner awaits the missing piece of knowledge from the teacher. Better yet, the learner starts developing their own answers. The learner craves knowledge.

Code puzzles open an immediate knowledge gap. When looking at the code, you first do not understand what the puzzle means. The puzzle’s semantics are hidden. But only you can transform the unsolved puzzle into a solved one. Look at this riddle:

“What pulls you down and never lets go?”

Can you feel the tension? Opening and closing a knowledge gap is a very powerful method for effective learning.

(By the way, the answer is Gravity.)

Bad teachers open a knowledge gap that is too large. The learner feels frustrated because they cannot overcome the gap. Suppose you are standing before a river that you must cross. But you have not learned to swim, yet. Now, consider two rivers. The first river is the Colorado River that carved out the Grand Canyon–quite a gap. The second river is Rattlesnake Creek. The fact that you have never heard of this river indicates that it is not too big of an obstacle. Most likely, you will not even attempt to swim through the big Colorado River. But you could swim over the Rattlesnake if you stretch your abilities just a little bit. You will focus, pep-talk yourself, and overcome the obstacle. As a result, your swimming skills and your confidence will grow a little bit.

Puzzles are like the Rattlesnake–they are not too great a challenge. You must stretch yourself to solve them, but you can do it if you go all-out.

Constantly feeling a small but non-trivial knowledge gap creates a healthy learning environment. Stretch your limits, overcome the knowledge gap, and become better–one puzzle at a time.

2. Embrace the Eureka Moment

Humans are unique because of their ability to learn. Fast and thorough learning has always increased our chances of survival. Thus, evolution created a brilliant biological reaction to reinforce learning in your body.

Your brain is wired to seek new information; it is wired to always process data, to always learn.

Did you ever feel the sudden burst of happiness after experiencing a eureka moment? Your brain releases endorphins, the moment you close a knowledge gap. The instant gratification from learning is highly addictive, but this addiction makes you smarter. Solving a puzzle gives your brain instant gratification. Easy puzzles open small, hard puzzles, which open large knowledge gaps. Overcome any of them and learn in the process.

3. Divide and Conquer

Learning to code is a complex task. You must learn a myriad of new concepts and language features. Many aspiring coders are overwhelmed by the complexity. They seek a clear path to mastery.

People tend to prioritize specific activities with clearly defined goals. If the path is not clear, we tend to drift away toward more specific paths.
Most aspiring coders think they have a goal: becoming a better coder. Yet, this is not a specific goal at all. So what is a specific goal? Unfortunately for many of us, the goal “Watching Game of Thrones after dinner, Series 2 Episode 1” is as specific as it can be. Due to the specificity, watching Netflix is more powerful than the fuzzy path of learning to code. Hence, watching Netflix wins the battle for attention too often.

As any productivity expert tells you: break a big task or goal into a series of smaller steps. Finishing each tiny step brings you one step closer to your big goal. Divide and conquer makes you feel in control, pushing you one step closer toward mastery. You want to become a master coder? Break the big coding skill into a list of sub-skills–understanding language features, designing algorithms, reading code–and then tackle each sub-skill one at a time.

Code puzzles do this for you. They break up the huge task of learning to code into a series of smaller learning units. The student experiences laser focus on one learning task such as recursion, the for loop, or keyword arguments. Don’t worry if you do not understand these concepts yet–after working through this book, you will. A good code puzzle delivers a single idea from the author’s into the student’s head. You can digest one puzzle at a time. Each puzzle is a step toward your bigger goal of mastering computer science. Keep solving puzzles and you keep improving your skills.

4. Improve From Immediate Feedback

As a child, you learned to walk by trial and error–try, receive feedback, adapt, and repeat. Unconsciously, you will minimize negative and maximize positive feedback. You avoid falling because it hurts, and you seek the approval of your parents. But not only organic life benefits from the great learning technique of trial and error. In machine learning, algorithms learn by guessing an output and adapting their guesses based on their correctness. To learn anything, you need feedback such that you can adapt your actions.

However, an excellent learning environment provides you not only with feedback but with immediate feedback for your actions. In contrast, poor learning environments do not provide any feedback at all or only with a large delay. Examples are activities with good short-term and bad long-term effects such as smoking, alcohol, or damaging the environment. People cannot control these activities because of the delayed feedback. If you were to slap your friend each time he lights a cigarette–a not overly drastic measure to safe his life–he would quickly stop smoking. If you want to learn fast, make sure that your environment provides immediate feedback. Your brain will find rules and patterns to maximize the reinforcement from the immediate feedback.

The Finxter App offers you an environment with immediate feedback to make learning to code easy and fast. Over time, your brain will absorb the meaning of a code snippet quicker and with higher precision this way. Learning this skill pushes you toward the top 10% of all coders. There are other environments with immediate feedback, like executing code and checking correctness, but puzzle-based learning is the most direct one: Each puzzle educates with immediate feedback.

5. Measure Your Skills

You need to have a definite goal to be successful. A definite goal is a powerful motivator and pushes you to stretch your skills constantly. The more definite and concrete it is, the stronger it becomes. Holding a definite goal in your mind is the first and foremost step toward its physical manifestation. Your beliefs bring your goal into reality.

Think about an experienced Python programmer you know, e.g., your nerdy colleague or class mate. How good are their Python skills compared to yours? On a scale from your grandmother to Bill Gates, where is your colleague and where are you? These questions are difficult to answer because there is no simple way to measure the skill level of a programmer.
This creates a severe problem for your learning progress: the concept of being a good programmer becomes fuzzy and diluted. What you can’t measure, you can’t improve. Not being able to measure your coding skills diverts your focus from systematic improvement. Your goal becomes less definite.

So what should be your definite goal when learning a programming language? To answer this, let us travel briefly to the world of chess, which happens to provide an excellent learning environment for aspiring players. Every player has an Elo rating number that measures their skill level. You get an Elo rating when playing against other players–if you win, your Elo rating increases. Victories against stronger players lead to a higher increase of the Elo rating. Every ambitious chess player simply focuses on one thing: increasing their Elo rating. The ones that manage to push their Elo rating very high, earn grand master titles. They become respected among chess players and in the outside world.

Every chess player dreams of being a grandmaster. The goal is as definite as it can be: reaching an Elo of 2400 and master level. Thus, chess is a great learning environment–every player is always aware of their skill level. A player can measure how decisions and habits impact their Elo number. Do they improve when sleeping enough before important games? When training opening variants? When solving chess puzzles? What you can measure, you can improve.

The main idea of our book Coffee Break Python, and the associated learning app, is to transfer this method of measuring skills from the chess world to programming. Suppose you want to learn Python. The Finxter website assigns you a rating number that measures your coding skills. Every Python puzzle has a rating number as well, according to its difficulty level. You play against a puzzle at your difficulty level: The puzzle and you will have more or less the same Elo rating so that you can enjoy personalized learning. If you solve the puzzle, your Elo increases and the puzzle’s Elo decreases. Otherwise, your Elo decreases and the puzzle’s Elo increases. Hence, the Elo ratings of the difficult puzzles increase over time. But only learners with high Elo ratings will see them. This self-organizing system ensures that you are always challenged but not overwhelmed, while you constantly receive feedback about how good your skills are in comparison with others. You always know exactly where you stand on your path to mastery.

6. Individualized Learning

The educational system today is built around the idea of classes and courses. In these environments, all students consume the same learning material from the same teacher applying the same teaching methods.
This traditional idea of classes and courses has a strong foundation in our culture and social thinking patterns. Yet, science proves again and again the value of individualized learning. Individualized learning tailors the content, pace, style, and technology of teaching to the student’s skills and interests. Of course, truly individualized learning has always required a lot of teachers. But paying a high number of teachers is expensive (at least in the short term) in a non-digital environment.

In the digital era, many fundamental limitations of our society begin to crack.
Compute servers and intelligent machines can provide individualized learning with ease. But with changing limitations, we must adapt our thinking as well.
Machines will enable truly individualized learning very soon; yet society needs time to adapt to this trend.

Puzzle-based learning is a perfect example of automated, individualized learning. The ideal puzzle stretches the student’s abilities and is neither boring nor overwhelming. Finding the perfect learning material for each learner is an important and challenging problem. Finxter uses a simple but effective solution to solve this problem: the Elo rating system. The student solves puzzles at their individual skill level. Our programming textbook “Coffee Break Python” and the book’s web backend Finxter pushes teaching toward individualized learning.

7. Small is Beautiful

The 21st century has seen a rise in microcontent. Microcontent is a short and accessible piece of valuable information such as the weather forecast, a news headline, or a cat video. Social media giants like Facebook and Twitter offer a stream of never-ending microcontent. Microcontent is powerful because it satisfies the desire for shallow entertainment. Microcontent has many benefits: the consumer stays engaged and interested, and it is easily digestible in a short time. Each piece of microcontent pushes your knowledge horizon a bit further. Today, millions of people are addicted to microcontent.

However, this addiction will also become a problem to these millions. The computer science professor Cal Newport shows in his book “Deep Work” that modern society values deep work more than shallow work. Deep work is a high-value activity that needs intense focus and skill. Examples of deep work are programming, writing, or researching. Contrarily, shallow work is every low-value activity that can be done by everybody (e.g., posting the cat videos to social media). The demand for deep work grew with the rise of the information society; at the same time, the supply stayed constant or decreased, e.g., because of the addictiveness of shallow social media. People that see and understand this trend can benefit tremendously. In a free market, the prices of scarce and demanded resources rise. Because of this, surgeons, lawyers, and software developers earn $100,000 per year and more. Their work cannot easily be replaced or outsourced to unskilled workers. If you are able to do deep work, to focus your attention on a challenging problem, society pays you generously.

What if we could marry the concepts of microcontent and deep work?

This is the promise of puzzle-based learning. Finxter offers a stream of self-contained microcontent in the form of hundreds of small code puzzles. But instead of just being unrelated microcontent, each puzzle is a tiny stimulus that teaches a coding concept or language feature. Hence, each puzzle pushes your knowledge in the same direction.

Puzzle-based learning breaks the bold goal, i.e., reach the mastery level in Python, into tiny actionable steps: solve and understand one code puzzle per day. While solving the smaller tasks, you progress toward your larger goal.
You take one step at a time to eventually reach the mastery level.

A clear path to success.

8. Active Beats Passive Learning

Robust scientific evidence shows that active learning doubles students’ learning performance. In a study on that matter, test scores of active learners improve by more than one grade compared to their passive learning fellow students. Not using active learning techniques wastes your time and hinders you in reaching your full potential in any area of life. Switching to active learning is a simple tweak that will instantly improve your performance when learning any subject.

How does active learning work?

Active learning requires the student to interact with the material, rather than simply consuming it. It is student- rather than teacher-centric. Great active learning techniques are asking and answering questions, self-testing, teaching, and summarizing. A popular study shows that one of the best learning techniques is practice testing. In this learning technique, you test your knowledge even if you have not learned everything yet. Rather than “learning by doing”, it’s “learning by testing”.

However, the study argues that students must feel safe during these tests. Therefore, the tests must be low-stake, i.e., students have little to lose. After the test, students get feedback about the correctness of the tests. The study shows that practice testing boosts long-term retention of the material by almost a factor of 10. As it turns out, solving a daily code puzzle is not just another learning technique–it is one of the best.

Although active learning is twice as effective, most books focus on passive learning. The author delivers information; the student passively consumes the information. Some programming books include active learning elements by adding tests or by asking the reader to try out the code examples. Yet, I always found this impracticable while reading on the train, on the bus, or in bed. But if these active elements drop out, learning becomes 100% passive again.

Fixing this mismatch between research and common practice drove me to write this book about puzzle-based learning. In contrast to other books, our programming textbook makes active learning a first-class citizen. Solving code puzzles is an inherent active learning technique. You must develop the solution yourself, in every single puzzle. The teacher is as much in the background as possible–they only explain the correct solution if you couldn’t work it out yourself. But before telling you the correct solution, your knowledge gap is already ripped wide open. Thus, you are mentally ready to digest new material.

To drive this point home, let me emphasize this argument again: puzzle-based learning is a variant of the active learning technique named practice testing. Practice testing is scientifically proven to teach you more in less time.

9. Make Source Code a First-class Citizen

Each grandmaster of chess has spent tens of thousands of hours looking into a near infinite number of chess positions. Over time, they develop a powerful skill: the intuition of the expert. When presented with a new position, they are able to name a small number of strong candidate moves within seconds. They operate on a higher level than normal people. For normal people, the position of a single chess piece is one chunk of information. Hence they can only memorize the position of about six chess pieces. But chess grand masters view a whole position or a sequence of moves as a single chunk of information. The extensive training and experience has burned strong patterns into their biological neural networks. Their brain is able to hold much more information–a result of the good learning environment they have put themselves in.

What are some principles of good learning?

Let us dive into another example of a great learning environment–this time for machines. Recently, Google’s artificial intelligence AlphaZero has proven to be the best chess playing entity in the world. AlphaZero uses artificial neural networks. An artificial neural network is the digital twin of the human brain with artificial neurons and synapses. It learns by example much like a grandmaster of chess. It presents itself a position, predicts a move, and adapts its prediction to the extent the prediction was incorrect.

Chess and machine learning exemplify principles of good learning that are valid in any field you want to master. First, transform the object to learn into a stimulus that you present to yourself over and over again. In chess, study as many chess positions as you can. In math, make reading mathematical papers with theorems and proofs a habit. In coding, expose yourself to lots of code. Second, seek feedback. Immediate feedback is better than delayed feedback. However, delayed feedback is still much better than no feedback at all. Third, take your time to learn and understand thoroughly. Although it is possible to learn on-the-go, you will cut corners. The person who prepares beforehand always has an edge. In the world of coding, some people recommend learning by coding practical projects and doing nothing more. Chess grandmasters, sports stars, and intelligent machines do not follow this advice. They learn by practicing isolated stimuli again and again until they have mastered them. Then they move on to more complex stimuli.

Puzzle-based learning is code-centric. You will find yourself staring at the code for a long time until the insight strikes. This creates new synapses in your brain that help you understand, write, and read code fast. Placing code in the center of the whole learning process creates an environment in which you will develop the powerful intuition of the expert.

Maximize the learning time you spend looking at code rather than at other stimuli.

10. What You See is All There is

My professor of theoretical computer science used to tell us that if we only stare long enough at a proof, the meaning will transfer into our brains by osmosis. This fosters deep thinking, a state of mind where learning is more productive. In my experience, his staring method works—but only if the proof contains everything you need to know to solve it. It must be self-contained.

A good code puzzle beyond the most basic level is self-contained. You can solve it purely by staring at it until your mind follows your eyes—your mind develops a solution based on rational thinking. There is no need to look things up. If you are a great programmer, you will find the solution quickly. If not, it will take more time but you can still find the solution—it is just more challenging.

My gold standard was to design each puzzle such that it is mostly self-contained. However, to deliver on the book’s promise of training your understanding of the Python basics, puzzles must introduce syntactical language elements as well. But even if the syntax in a puzzle challenges you, you should still develop your own solutions based on your imperfect knowledge. This probabilistic thinking opens the knowledge gap and prepares your brain to receive and digest the explained solution. After all, your goal is long-term retention of the material.

What do you think of puzzle-based learning? Please share your thoughts and leave a comment below.

The post 10 Reasons Why Solving Code Puzzles Makes You Smarter appeared first on Be on the Right Side of Change.