The following problem formulation is taken from this Kata.

Design a recursive function called replicate(times, number) that returns an array of repetitions of the number argument.

For example, replicate(3, 5) should return the list [5, 5, 5].

If the times argument is negative, return an empty array.

Do not use loops to solve this problem.

Background Resources

• Recursion: A Helpful Guide in Python
• Python One Line Recursion

Video Solution

This video solution is contributed by Finxter Creator Clement:

Code Solution

Here’s the code discussed in tihs video:

def replicate(times, number):
    # the forbidden way
    result = []
    for i in range(times):
        result.append(number)
    return result


def replicate(times, number):
    # recursive solution
    if times <= 0:
        return [] # base case and exception handling
    return replicate(times-1, number) + [number]
    
    

The first function uses the for loop, the second function uses the recursive solution with the base case and the recursion case.

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

The post Kata Recursive Replication – Python Solution Video appeared first on Be on the Right Side of Change.

Hey Finxters!

In my last article, I showed you how to host your Streamlit/Python web app in the cloud on Heroku.

Now, perhaps you thought to yourself, that’s very nice but I need my script (not necessarily a web app, by the way) to run twice a week to load this new data at work, or to send an email to these customers every night at 3.00 a.m European time, or perhaps as a hobby, you’d like to get in your emailbox that football data freshly scraped overnight?

If so, then keep reading!

While there are many platforms allowing you to do this starting at 0€/month (AWS, Google, Python Anywhere, cron, for example), we will stick with Heroku in this tutorial, to be consistent with my previous article.

This article assumes you have:

• basic knowledge of Linux command line
• read my previous article on pushing Python code on Heroku (recommended)
• the Heroku CLI installed
• basic Git notions
• a Heroku account (free) (do not forget to configure your email address)
• basic Python notions

Plan for today:

Prepare your Python script

In this tutorial, I need a script in which the result is visible so that you can actually view the output, so I decided on … wait for it… sending an email with a scheduled agenda, as this will provide a very visual way to check the results and their timing.

So here I went, creating the folder and starting to populate it with the necessary files: 

Then scripting this snippet:

Now adding the credentials.json and token.json so that the script can access the Gmail account. (Need a refresher? => See here):

As a reminder:

• use a toy email account if this is for hobby
• don’t push such sensitive files as credentials.json to public repos. (By the way, there is this great startup, Gitguardian, which job is to detect such risks and alert accordingly)

Push it on Heroku

Now you must be accustomed to it, so here I go:

Notice I did not specify any app name (heroku create), so Heroku will find an available name randomly. This is because it doesn’t really matter as this is not a website, but only a scheduled process running in the background, so noone will see its name except me (and potential collaborators).

What a great name though! : Enigmatic escarpment!

What happens if I click on the URL provided with the project?

Oouups! Did I mess things up?

No! That’s normal, Heroku provides me with a URL but I’m not making a web app this time.

Trigger a Scheduler

Now that our script is in the cloud, nothing will happen until we select and trigger a scheduler. We will do it the GUI way.

To do so, go to your app page, click on the Resources tab, and start typing ‘schedule’ in the add-ons bar and you will see several schedulers appear:

Depending on your goal and your budget, you may pick one of the above.

I’ll go with the free plan of the advanced scheduler for this tutorial.

Just select the one that best suits your need, and click on submit order form.

Then click on the scheduler to start setting it, and you will see a purple button ‘Create first trigger’, click on it.

At this stage, you might get a pop-up requiring you to fill in your bank details before being able to proceed. If you then opt for a free plan, you can do so without worry of being charged.

You will be prompted to type a name for the trigger, the command to run.

The command is the one you would type in your shell to run your Python program (in this project, the command to type is thus: python scheduled_script.py). 

Then select your timezone and whether this is a one-off or recurring.

In my case and for this tutorial, I selected recurring every minute:

Finally, you can monitor your scheduler add-on on the dedicated page that follows:

Enjoy your automation!

You can now sit and relax, while Python on Heroku is doing all the nightly work for you!

You remember I wrote and pushed a program that sends emails to myself every single minute, with the time in the subject of the email? 

Look at the results:

Just imagine the endless possibilities for your future programs!

Beware though, I do not recommend using a free plan for serious projects, as the reliability might not be up to standards and your dyno might run out.

Also, please use this kind of technology responsibly.

That’s all for today, I’m sure you can now try it yourself and create great things.

Where to Go From Here?

• Creating your own scheduled Python task
• try out other schedulers
• try out the Advanced Scheduler Service API

The post How to Schedule Your Python Script for Free on Heroku appeared first on Be on the Right Side of Change.

• Have you ever felt surrounded by developers boasting from their latest app in prod hosted in the cloud? 
• Or the need for making yours too but overwhelmed by the technicalities involved?
• Needed to set up quickly and easily a mini data science demo website without resorting to web developers?
• Or simply wanting to share your Python work with non-technical people?

Then this article is for you.

? Today I will show you one of the simplest ways to set-up your very own Python web app in the cloud, using streamlit to design the app together with Heroku for the hosting part. And it is free.

This article assumes you have at least:

• basic Python notions
• basic Git notions
• a Heroku account (free) (do not forget to configure your email address)
• the Heroku CLI installed
• basic notions of Linux commands
• basic notions of bash scripting

Ready? Here is the plan of our journey:

What is Streamlit?

Streamlit is a data science oriented application framework. 

Its goal is to enable data scientists to release applications without requiring the assistance of a development team

Streamlit allows for example to build an app making predictions using a trained machine learning model, with very few lines of code handling the user interface and controls, and almost no design efforts.

During this tutorial, I will be referring heavily to the Streamlit documentation, and I strongly encourage you to spend some time reading it.

To use it you will first need to do the following in your terminal:

pip install streamlit
streamlit hello  # not strictly mandatory; launches the ‘hello world’ app to test it works

Then in your Python file, the canonical import will be:

import streamlit as st

What is Heroku?

From their own words: “Heroku is a cloud platform that lets companies build, deliver, monitor and scale apps — we’re the fastest way to go from idea to URL, bypassing all those infrastructure headaches.”

That’s right, at the end of this article, we will get an actual URL for our Python project!

Because we will be using a free account, the URL will look like: our_project_name.herokuapp.com, but you might want to upgrade in order to have more options.

So in a nutshell, we will be hosting our Python web app designed with Streamlit on their server, in order for other people to access it without having to have our machine running 24/7.

More info on Heroku can be found here.

Building the Project

In order to keep things as simple as possible, I will show you the minimum package needed to successfully run your app, but of course you may encounter additional files in other Streamlit/Heroku repos depending on what is achieved, such as for instance:

.gitignore, README.md, Makefile, MANIFEST.in, setup.py, runtime.txt, credentials.json, token.json, some_machine_learning_trained_model.h5, etc.

I will not comment on these files specifics in this tutorial, as we will focus on building and hosting our Python web app.

Start by creating a folder where you will put all the necessary files:

mkdir my_project_folder && cd my_project_folder

If you already have a Python project coded, say on Github: then git clone it.

Here is the list of files that we will need in our folder before pushing to Heroku, I will go over each of them:

• main.py 
• requirements.txt
• Procfile
• setup.sh

And that’s all! Let’s dive in the files specifics:

main.py

This is the Python file (you may need several depending on your project).

This is where your Python code lies and also where the web app structure comes in, thanks to streamlit. Remember you imported streamlit as st at the top of this file?

You can now design many elements thanks to this, for example:

• markdown : st.markdown('''some text ''')
• a slider  : st.slider(*args, *kwargs)
• a file uploader : st.file_uploader(*args, *kwargs)

as well as many, many more features, developed by an active community.

One more tip: to tailor things such as your app’s favicon, title etc., look into st.set_page_config and place it at the top of your Python script, just below the imports. 

You should have a look at the documentation to get a grasp of the possibilities there.

Finally, if you fancy looking at sample websites, visit the Streamlit gallery!

Please note that every time the user will interact with your app, the code will be rerun from top to bottom. This may result in unresponsiveness at times.

requirements.txt

This is the text file that specifies which (not built-in) modules will be needed for your web app to run on Heroku.

The minimum it should contain is therefore streamlit.

It basically just lists the (not built-in) modules imported in your Python file(s) for this project.

Beware the 3 following items:

• version constraints must be stated (ex: scikit-learn>=0.21.3 or requests==2.*) 
• as well as the exact module name (the name may differ between what is pip installed and what is imported)
• as we are using a free Heroku plan, the slug size is limited to 500mB thus preventing heavy modules, such as Tensorflow

touch requirements.txt then add manually the modules, or use a script that does it for you, or echo “modules” > requirements.txt

if you setup a virtual env for your app, they can be obtained through a pip freeze

Procfile

That’s right, this file has no extension. It is a process file specific to Heroku that details which commands should be executed by the app on startup. In our case a web process. More details here

For now, you can simply echo > the following snippet in your Procfile:

web: sh setup.sh && streamlit run <your_python_file>.py

Of course, you should adapt it with your own Python file’s name (main.py in this tutorial).

setup.sh

Custom script launched by the Procfile command. Indicates to Streamlit the app owner’s email and Heroku’s dynamically assigned port on which to run.

Like before, for now you can just copy this piece of code:

mkdir -p ~/.streamlit/

echo "\
[general]\n\
email = \"${HEROKU_EMAIL_ADDRESS}\"\n\
" > ~/.streamlit/credentials.toml

echo "\
[server]\n\
headless = true\n\
enableCORS = false\n\
port = $PORT\n\
" > ~/.streamlit/config.toml

And that’s it! Now that you have the full structure in mind, let’s see an example.

Perhaps you’ve read my article about sending email from a Gmail account with Python?

Well, I actually made a Streamlit/Heroku web app with it, and here is the resulting directory:

Testing the App Locally

To check that your Python file works and make modifications to it quickly, use the following command to run it locally:

streamlit run <your_python_file>.py

Either your browser will open and show your app, or two URLs will appear, allowing you to view your app. Click on Network URL.

Here is what I’d get in my browser in my example email project:

If you want to have a closer look, find my repo here.

Pushing the Project

Last but not least, we now need to make our web app available to the world, as for now it only runs locally.

How to proceed?

Like so:

# if not already done, log in to Heroku
heroku login

# create the app in your region and pick an available APP_NAME
heroku create APP_NAME --region eu

An example for say, a Finxter web app, would go:

# you should see the origin remote for GitHub if you cloned a repo
# and the heroku remote for Heroku
git remote -v

Example from my terminal:

# manually add the heroku remote if necessary
git remote add heroku https://git.heroku.com/APP_NAME.git

# if not already done
git add . # or specify files if you don’t want to push everything
git commit -m “<your commit comment>”

# deploy the app 
git push heroku master  # this could take up to a few minutes and you will witness the build. Change the branch name if needed (eg “main” instead of “master”)

If you read the logs above, you should now be able to understand the deployment process.

# start the web dyno (this is not strictly necessary in most cases)
heroku ps:scale web=1

# open a browser to your deployed web app!
heroku open

Now, you may experience issues at deployment stage.

Hence a useful command to know is:

# check the logs for errors
heroku logs --tail  
# will help you troubleshoot, by showing the build logs (you can also view them on your Heroku app profile on Heroku’s website)

The Heroku GUI in your profile will look like this:

Last, a few remarks:

• this requires a bit of practice before deploying with ease but is definitely worth it!
• with a free plan, expect your app to sleep after some time of inactivity. I found a possible remedy to this here
• whenever your app is awakened, it may take up to 30 seconds to load, that is perfectly normal
• with a free, unverified Heroku account, you can have up to 5 apps running at the same time
• the free plan allocates some number of dyno hours per month after which your app will sleep
• despite all these limitations, you can still very much present data science results or any Python work really and get a reliable app!

I hope you enjoyed our journey into the Python web app deployment universe and that you will create amazing apps!

Oh wait! I said I deployed my email sender project, fancy seeing it?

Here it is: https://email-sender-finxter.herokuapp.com/

You can add several recipients as well as add attachments.

With great power comes great responsibility: use it responsibly.

Where to Go From Here

• make your own Python web app!
• tune it
• maybe onboard a trained machine learning model on it?
• share it to the world!
• filter email content with NLP

I hope you will benefit from my experience on this topic, see you next time guys!

The post How to Build and Host Your Python Web App for Free appeared first on Be on the Right Side of Change.

Project Goal

Project goal: code your own Google Home with Python and learn how to use speech recognition to launch Spotify and play songs!

Ever wanted to code a powerful yet simple tool that is more bespoke than mainstream devices?

We will learn how to implement it in Python with a bunch of powerful libraries!

Breaking down our problem, there are three tasks ahead of us:

• processing speech and convert it into text
• based on some string condition in the text, open a process (here, the Spotify app)
• interact with the process

Performing Speech Recognition

Don’t do the heavy lifting yourself!

Speech recognition is the ability to detect and identify words and phrases in spoken language and subsequently convert them into human, readable text. 

This field can be very complex and top Python libraries result from decades of hard work from experts. We will obviously not be constructing from A to Z such a library, that would be way beyond this tutorial. Instead, we will be using the SpeechRecognition library.

Therefore, we don’t need to build any machine learning model from scratch, this library provides us with wrappers for several well known public speech recognition APIs (such as Google Cloud Speech API, IBM Speech To Text, etc.).

As usual, we will start downloading the module:

pip install SpeechRecognition pydub

Then, in a new Python file, you can import it the following way:

import speech_recognition as sr

It is now very handy, you have access to several speech recognition engines, that have different use cases:

• CMU Sphinx (offline)
• Google Speech Recognition
• Google Cloud Speech API
• Wit.ai
• Microsoft Bing Voice Recognition
• Houndify API
• IBM Speech To Text
• Snowboy Hotword Detection (offline)

In this tutorial, we will be using Google Speech Recognition, because it is rather simple to use, efficient, and does not require any API key.

Interpreting Speech from a File

Regular Size

Before starting, make sure you placed an audio file containing english language, in the current working directory for maximised simplicity, or somewhere you know the path (such as ‘../audio_files/my_audio_file.wav’).

The first step is to initialize your recognizer like so:

# initialize the recognizer
r = sr.Recognizer()

The below code is then responsible for loading the audio file from the designated path, and then converting the speech into text using Google Speech Recognition:

# open the file
with sr.AudioFile(path_to_audio_file) as source:
    # listen to the data ( = load audio to memory)
    audio_data = r.record(source)
    # recognize (convert from speech to text)
    text = r.recognize_google(audio_data)
    print(text)

 This could take a little while, on the other hand don’t think the duration of the code execution is in some way related to human speech speed: you will commonly eye-witness your code spitting out the full text even before the audio file has finished reading!

Okay, this kind of script works fine for small to medium-sized audio files, not so well for larger files.

Large Audio Files

I won’t go into too many details here, as our goal is to launch Spotify thanks to voice command, remember? Suggesting we’ll use the mic.

However, if you do need to convert the content of large audio files, then you should be looking into the pydub library, more specifically its AudioSegment class and split_on_silence function.

Why?

Because, well-equipped with these two, you will then be able to respectively load the audio data and chunk it, based on a pre-set silence duration found in the data.

This comes handy to split your audio file.

Recommended: How to Recognize Speech From Large Audio Files?

Interpreting Speech from the Mic

We are now getting to the core of the tutorial! We will be processing the audio input directly from the mic, moving one step closer to being able to actually make voice commands.

To start with, this requires PyAudio to be installed on your machine and depending on your OS the installation procedure varies:

Windows

pip install pyaudio

Linux

You need to first install the dependencies:

sudo apt-get install python-pyaudio python3-pyaudio
pip install pyaudio

MacOS

You need to first install portaudio:

brew install portaudio
pip install pyaudio

Warning: you may experience issues installing the module properly, especially on Windows.

For Windows users, if you don’t succeed the abovementioned way, try:

pip install pipwin (if you don’t already have it)
pipwin install pyaudio

Now we’re ready to start building our Spotify launcher!

with sr.Microphone() as source:
    # read the audio data from the microphone
    audio_data = r.record(source, duration=5)
    print("Analyzing...")
    # convert speech to text
    text = r.recognize_google(audio_data)
    print(text)

This piece of code will open the (default) mic, read the input for 5 seconds (you can obviously tailor this argument), then (try to) convert it, finally print out the output. 

Obviously it still isn’t perfect, for example it usually struggles with homophonous phrases or words.

Two arguments are worth mentioning at this point:

• offset: passed to the record function, it is used to start recording after some delay (default 0)
• language: passed to the recognize_google function, it changes the target language (ex: “fr-FR”). More info about supported languages here

Opening Processes with Python

Now that we can talk to our speech recognizer and convert speech into text, let’s head towards our second task: opening processes.

As often, there are multiple ways to do this.

We will use the subprocess module, which is built-in.

In particular, we will be using the Popen (P stands for process) function within this module, like so: 

# import the function from its module
from subprocess import Popen

Popen(path_of_the_executable)

For example, on my machine, opening the Spotify desktop app would be done like so:

subprocess.Popen('C:\\Users\\cleme\\AppData\\Roaming\\Spotify\\Spotify.exe')

Depending on your OS, you may need to adjust the slashes in the path to make sure it is understood well. You may want to use a function doing this for you in the os built-in module.

Of course, as always feel free to dive deeper into this module, but for now we have what we need to trigger the opening of our Spotify (desktop) app.

Interacting with Windows

OK, let us sum up:

• we know how to convert speech to text
• we know how to open processes

From there, we can easily create a condition on the output text from the speech conversion; for example :

if "spotify" in text.lower():
    subprocess.Popen('C:\\Users\\cleme\\AppData\\Roaming\\Spotify\\Spotify.exe')

What is there left to do?

Now that our Spotify app is open on voice command, we need to somehow be able to launch a song.

To do this, there may exist Spotify modules, but we will be using a powerful module:

pyautogui

This module is essentially about automating mouse, windows, and keyboard actions!

So, what we’ll do is identify the Spotify app search bar location, click it, clear it if needed, type a song or artist name in it, then press Enter, then press Play, and we’re done!

pip install pyautogui

The first thing is to make sure we are dealing with the Spotify app window.

To do this we’ll loop over pyautogui.getAllWindows(), which yields all the currently opened windows titles, and make an if statement to select the Spotify window.

We will then proceed to the subtasks identified above.

We will use a convention here in our voice command: for the sake of simplicity, we will assume that the name of the wanted artist comes last in the voice command (e.g.: “Please open Spotify and play Madonna”).

Of course this is a dummy example, but you can easily improve the voice command and make it more flexible.

Here is what it looks like:

    for window in pyautogui.getAllWindows():
        if 'spotify' in window.title.lower():
            window.show()
            print('spotify window activated')
            text = text.split()  # break down text into list of single words strings, for later usage
            time.sleep(5.5)
            pyautogui.click(x=480,y=25) # this is the search bar location on my machine, when the window is maximized
            time.sleep(1)
            pyautogui.hotkey('ctrl','a') # clearing the search bar
            pyautogui.press('backspace') # clearing the search bar
            time.sleep(0.5)
            pyautogui.write(text[-1],interval=0.05) # because we assumed that the artist was the last word of the voice command
            time.sleep(3)
            pyautogui.click(x=380,y=250) # this is the play button location on my machine, when the window is maximized
            break

Breaking down this piece of code, we performed sequentially all the steps we identified as mandatory. 

Note the difference between the hotkey (keystroke combination) and press (single key, down then up) methods.

We used Ctrl+a to select all potential text in the search bar, then we removed it before typing our artist’s name. The text[-1] bit refers to the last word of our voice command, see the convention described above.

Please note the interval argument inside the write method: in some cases it is vital for our script to function properly. Why?

Because it is the argument that sets the typing speed, and in some instances, pyautogui just goes too fast for the process and this ends up in an unwanted result.

You might need to manually finetune this argument, like I did before settling with 0.05. In the same vein, the time.sleep() statements are here to make sure our code does not outspeed too much the app, for example it enables to wait for the proper opening of the app. This can imply some manual tries.

Last, the break statement is there to make sure we go out of the for loop once we have found our app. Let’s not waste time checking useless windows!

Alright, we’re almost there, I can hear the song now!

Now you may wonder, what if we need to stop the song from playing?

Well, we’ll take care of just that in the below piece of code:

while True:
        try:
            with sr.Microphone() as source:
                # read the audio data from the default microphone
                audio_data = r.record(source, duration=1)
                # convert speech to text
                text = r.recognize_google(audio_data)
                if 'stop' in text.lower():
                    pyautogui.click(x=955 ,y=1000)
                    break
        except Exception as e:
            continue
          print(f"Encountered error: {e}\n")

The while True loop is there to keep looping until it hears ‘stop’ (again, you can obviously tailor this criterion).

If ‘stop’ is heard and correctly decoded, then pyautogui presses the Stop button for us. (Do not hesitate to look into parameters that enable to improve the mic detection when there is noise around (here, our song playing)).

With the use of a try/except clause, we are able to keep the program running without being bothered by potential errors in the way, but still able to print them if they appear, for later debugging.

Combining Everything

Want to see my full code? Here it is, below:

import pyautogui, subprocess, os, time
import speech_recognition as sr
 
# initialize the recognizer
r = sr.Recognizer()
 
with sr.Microphone() as source:
    # read the audio data from the default microphone
    audio_data = r.record(source, duration=3)
    print("Recognizing...")
    # convert speech to text
    text = r.recognize_google(audio_data)
    print(f"I think you said: '{text}'\nhmmm, let's see what I can do for you.")
 
if "spotify" in text.lower():
    subprocess.Popen('C:\\Users\\cleme\\AppData\\Roaming\\Spotify\\Spotify.exe')
    for window in pyautogui.getAllWindows():
        if 'spotify' in window.title.lower():
            window.show()
            print('spotify window activated')
            text = text.split()  # break down text list into single words for later usage
            time.sleep(5.5)
            pyautogui.click(x=480,y=25) # this is the search bar location on my machine, when the window is maximized
            time.sleep(1)
            pyautogui.hotkey('ctrl','a') # clearing the search bar
            pyautogui.press('backspace') # clearing the search bar
            time.sleep(0.5)
            pyautogui.write(text[-1],interval=0.05) # because we assumed that the artist was the last word of the voice command
            time.sleep(3)
            pyautogui.click(x=380,y=250) # this is the play button location on my machine, when the window is maximized
            break
    while True:
        try:
            with sr.Microphone() as source:
                # read the audio data from the default microphone
                audio_data = r.record(source, duration=1)
                # convert speech to text
                text = r.recognize_google(audio_data)
                if 'stop' in text.lower():
                    pyautogui.click(x=955 ,y=1000)
                    break
        except Exception as e:
            continue
    

There is room for improvement but it works, you can be proud of you if you manage to code your own music launcher!

Thank you guys! That’s all for today.

Where to go from here?

• Create a voice command to change the sound level, or go to the next song
• Combine this script with Selenium to play music from the Internet instead
• Use machine learning for smarter interpretation of the voice command
• Convert text to speech
• Schedule tasks with Python
• In relation with the above item, look into cloud solutions to run code 24/7

The post Coding Your Own Google Home and Launch Spotify in Python appeared first on Be on the Right Side of Change.

Hey Finxters! Today we are going to talk about one of the most popular clustering algorithms: K-Means.

Ever wondered how to organize seemingly unstructured data, making sense of unordered objects, in an easy way?

For example, you might need to:

• perform customer segmentation
• store files based on their text content
• compress images with your own code

We will learn how to implement it in Python and get a visual output!

A Bit of Theory

In case you are not that much into theory and/or need to get working quickly, you can just skip this part and go to the next one.

First of all, the Machine Learning algorithm that we are about to learn is an unsupervised algorithm. What does that mean?

It means that we do not have beforehand any labels to use for the data-clustering, we might even have no idea what to expect! So in a way we are going to ask the algo to make groups where we might not necessarily see ones.

In addition to being unsupervised, we say this is a clustering algorithm because its point is to create sub-groups of datapoints that are close in some way, in terms of numerical distance. This idea was first implemented by the Bell labs in the late 1950s.

Perhaps the best way to view clusters for a human eye is in 3D like above, or in 2D; however, you rarely have so few features in the dataset. And it works better on data already clustered geometrically.

Which means it is often a good idea to start out reducing the dimensions, for example by means of a Principal Component Analysis algorithm.

Note that this algo must be assisted in that it requires the user to input the number of clusters to create. Each of them will have a center point called “centroid”.

Here is the procedure that will be run under the hood once we execute our code:

• Choose number of clusters K to look for (human input)
• Initialize K centroids randomly

• Compute mean-square distance of each datapoint with each centroid
• Assign each datapoint to the closest centroid (a cluster)
• Compute the mean of each cluster, which becomes your new centroids

The previous 3 steps make up what is called an epoch.

The program we will create will keep running epochs until centroids stop changing, i.e. convergence is obtained.

An image is worth a thousand words, so here is what it looks like:

Does K-means have a loss function?

Yes, it is called inertia and is the sum of squares of distances between data points and their respective centroids.

In practice

• K-means is usually run a few times with different random initializations
• Can use random mini-batch at each epoch instead of full dataset, for faster convergence
• Algorithm is quite fast

Installing the Module

The module that we will be using to perform this task is Scikit-Learn, a very handy module when it comes to Machine Learning in Python.

If you do not already have it, proceed with the usual install command:

pip install scikit-learn

Then, check it installed correctly:

pip show scikit-learn

Here is the sklearn documentation page dedicated to Kmeans: https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html#sklearn.cluster.KMeans , do not hesitate to check it for more details on the arguments you can pass and a more advanced use.

Once this is done, we will import the Kmeans class within this module:

The first line is the import.

Making the Magic Happen

The second line instantiates the Kmeans class by creating an actual Kmeans object, here is it put in a ‘km’ variable and the user asked for the creation of 3 clusters.

The third line launches the computation of the clustering.

Once your K-Means model is fitted, you can use four attributes that speak for themselves:

• km.cluster_centers_ : provides the coordinates of each centroid
• km.labels_ provides the cluster number of each datapoint (indexing starts at 0 like lists) 
• km.inertia_ : yields the sum of squared distances of samples to their closest centroid
• km.n_iter_ : provides the number of epochs run

If you want to try it but do not have a dataset ready, you can generate your own points thanks to sklearn make_blob feature!

Here is an example output in 2D, with a PCA dimensionality reduction as you can see on the x and y axes:

I showed you the attributes, what about the methods available?

The most useful one probably is the .predict(new_datapoint) method, that returns an integer corresponding to the cluster (number) estimated by the model.

How to Pick the Best Number of Clusters

Wait, this is all very nice if I know what to expect in terms of number of clusters, as i can then input this number, but what if I have no idea how many clusters to expect?

Then use the elbow method. It means graph the evolution of the inertia according to the number of clusters, and pick the number of clusters after which decrease in inertia becomes marginal:

In the above example, the ideal number of clusters seems to be 3. The graph is elbow-shaped, hence the name.

K-Means with NLP: Displaying a Wordcloud

Assuming you used a K-Means algorithm within a Natural Language Processing task, after preprocessing and vectorizing the words, you may be in need of a visual way to present your output.

Indeed, sometimes the number of clusters will be high and displaying labels in a grid will not be that impactful.

Then comes into play the wordcloud module, enabling you to generate easily pretty, colourful wordclouds for instant understanding.

Just pip install wordcloud and use 

plt.imshow( Wordcloud().generate(your_text) )

See documentation for parameters.

In my example shown above, I was dealing with Irish PDF reports, and in each report part of the content was written in Gaelic. 

Guess what the algo found? Look at the bottom cluster!

This illustrates the “unsupervised” characteristic: I did not tell it there was another language, and yet it found it and isolated it by itself!

Where to Go From Here?

I hope you enjoyed this article. To go deeper into the topics, do check the documentation and experiment yourself:

• PCA analysis
• More clustering techniques: https://scikit-learn.org/stable/modules/clustering.html
• More ideas to implement this algorithm : https://dzone.com/articles/10-interesting-use-cases-for-the-k-means-algorithm

Any comments? Let us know!

The post The K-Means Algorithm in Python appeared first on Be on the Right Side of Change.

Of course, many emails need your human understanding to be processed properly, and this article is not about implementing a neural network to fine-tune every single email action. 

However, it is about learning a key asset: how to implement a code to manage simple tasks on a Gmail account, with the ezgmail module, such as:  

• receiving and reading your unread messages 
• sending emails, including attachments 
• downloading attachments 
• viewing your recent messages 

Building on these simple tasks, you will then be able to endlessly tailor your mailbox according to your personal needs, such as for example: 

• trigger automatic replies based on specific strings found in an email body and/or subject,
• block a sender by throwing to the bin emails based on sender email address, or
• albeit more elaborate, make your very own spam detector by injecting an NLP algorithm in your program.

Lets get started! 

Setting It Up

For this project to work, you need two things: 

• installing the right module
• enabling the Gmail API

Install the ezgmail Module

The module well be working with is called ezgmail, and you can install it with the usual command: 

pip install ezgmail

or

pip3 install ezgmail

Alternatively, you may run

pip install --user -upgrade ezgmail

on Windows, to make sure you get the latest version. 

To check it installed correctly, you can then check the version with the following command line argument: 

pip show ezgmail 

Note that this module is not produced by or affiliated with Google. It was developed by software programer Al Sweigart and you can find all the details here: https://github.com/asweigart/ezgmail 

Enabling the Gmail API 

First of all, I highly recommend you setup a separate Gmail account for this project. This will prevent any unexpected event from altering your mailbox in any unwanted way. 

So, start out by signing up for a new Gmail account, then visit this page: https://developers.google.com/gmail/api/quickstart/python 

Click Enable the Gmail API button, then fill in the form, after which you’ll see a link to a credentials.json file, which you’ll have to download and place in the same directory as your Python file. This is a requirement for the code to work. (For those who don’t know, basically, json is a popular and widespread format that looks like a dictionary in Python.) 

Consider this file content the same as your Gmail password, so keep it secret. 

When you will run your Python code to manage your Gmail account, the code will first visit the json file directory in order to fetch your credentials from the credentials.json file. 

This provides additional security as unlike other modules, with ezgmail you do NOT have to type in plain text your credentials in the program. 

Time to start: enter the following code: 

import ezgmail, os
# change dir to the one where json credentials are saved:
os.chdir(r'C:/path_to_credentials_json_file')
ezgmail.init()

As a side remark, notice that you could have achieved what the second line does (i.e., changing the current working directory to the directory containing the file credentials.json) with the Python exec() function!

Something like:

exec(import os; os.system(cd path_to_credentials_json_file))  

The .init() method will open your browser towards a Google page where you can login. Type your credentials, you may then see This app isn’t verified: this is OK (believe me, I did it before you and I’m fine!), click Advanced, then Go to quickstart (unsafe). 

When the next page prompts you with Quickstart wants to access your Google account, allow it, then close the browser. 

You’re almost done with the setup phase.  

What just happened is a token.json file was created, and this will be used to provide your Python code access to the Gmail account you created for this project. Keep this one safe, too. 

So from now on, you will no longer need to manually type your credentials. 

You’re good to go! Starting now, the .init() method should no longer be necessary.  

Sending emails 

The method is quite straightforward: ezgmail.send() 

ezgmail.send('recipient@gmail.com', 'test', 'hello world!')

Here are the arguments that you can pass: 

Mandatory args are: 

• recipient 
• subject 
• body 

Optional, kwargs are: 

• attachment (you can pass a list if there are several) 
• sender 
• cc (might not work at the moment, as per the github page) 
• bcc (might not work at the moment, as per the github page) 

Forgot the email address the token.json was setup for? 

Just check the attribute ezgmail.EMAIL_ADDRESS  

ezgmail.EMAIL_ADDRESS

Receiving emails 

There are two steps involved:

• Reading the email, and
• Downloading its attachments.

Reading mail 

The ezgmail package structures emails just like the GUI email client does: it organizes them into threads, that can in turn contain multiple messages. 

Hence the method .unread() lists the GmailThread objects. 

print(ezgmail.unread())

Want to read a specific email within a thread? 

The .messages attribute is just what you need. It is subscriptable: 

unreadThreads = ezgmail.unread()
print(unreadThreads[0].messages[0].body)

It comes with a bunch of attributes such as sender, recipient, body, timestamp etc. 

Also check the .recent() method: it yields the 25 most recent threads of your Gmail account. 

recentThreads = ezgmail.recent()
print(ezgmail.summary(recentThreads))

Downloading attachments 

A GmailMessage object carries an attachments attribute that’s a list of filenames. 

Pass any combination of these filenames in the .downloadAttachment() method to download the files, or if you want all of them, use the .downloadAllAttachments() method, which even has an argument enabling  you to specify where to download the files (default the current working directory). 

Searching mail 

You guessed it use the ezgmail.search() method! 

Enter a string in this method just like you would in a GUI mailbox. 

resultThreads = ezgmail.search('json')
ezgmail.summary(resultThreads)

This returns a list of threads (remember the GmailThreads objects?) 

You can then pass the above mentioned attributes to retrieve specific info about a message. 

Where to go from here? 

Try it yourself! 

And do discover the other features provided by this efficient and user-friendly module! 

• Maybe you need it to automate a newsletter? 
• Or to setup email reminders for your personal need? 
• Or at work? 

Let us know! 

A few concluding remarks: 

• in general, for this to work the email account needs to be configured with the lowest level of security, else the email will end-up either blocked or in the spams 
• you may not be able to send repeated emails with the exact same text (as these probably are spams) nor with .exe attachments (since these are probably viruses) 
• please use this technology responsibly 
• thanks to Al Sweigart for creating and maintaining this awesome module 
• just because it currently works doesn’t mean it will forever; it is reliant on Googles choices among other things, and this module behaviour through time cannot be guaranteed 

Last, if you need to process emails from an account other than Gmail, you should check the right modules to send and receive emails from any account, respectively using SMTP and IMAP protocols. 

The post EZGmail and Python — Managing Your Emails Programmatically appeared first on Be on the Right Side of Change.