Step 2: Append “.json” to the URL

• Example Subreddit URL: https://www.reddit.com/r/GeminiAI/comments/1pyi6ax/how_to_tell_if_an_image_is_ai_generated
• Example JSON: https://www.reddit.com/r/GeminiAI/comments/1pyi6ax/how_to_tell_if_an_image_is_ai_generated.json

Step 3: Copy and paste the whole JSON data into ChatGPT/Gemini/Claude

Step 4: Prompt something like this to discover user needs:

Look at this Subreddit (JSON format). Find and list all user pain points, needs, and struggles. Suggest top products and unique selling propositions (USPs) to satisfy these needs. Research similar products and find unique positions.

Here’s a sample output on my ChatGPT 5.2 Thinking:

It also shows a number of products that would fill these needs:

I omit the rest for brevity – you get the point.

This truly is a goldmine of business need analysis.

The post The JSON Trick – How to Scrape All Answers in a Subreddit? (Example Find User Needs) appeared first on Be on the Right Side of Change.

You can’t parse [X]HTML with regex. Because HTML can’t be parsed by regex. Regex is not a tool that can be used to correctly parse HTML.

As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML.

Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions.

Regex queries are not equipped to break down HTML into its meaningful parts.

so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack.

HTML is a language of sufficient complexity that it cannot be parsed by regular expressions.

Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp.

Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide.

The <center> cannot hold it is too late.

The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the n​erves of the sentient whilst you observe, your psyche withering in the onslaught of horror.

Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the transgression of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of reg​ex parsers for HTML will ins​tantly transport a programmer’s consciousness into a world of ceaseless screaming, he comes, the pestilent slithy regex-infection wil​l devour your HT​ML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fi​ght he com̡e̶s, ̕h̵i​s un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo​͟ur eye͢s̸ ̛l̕ik͏e liq​uid pain, the song of re̸gular exp​ression parsing will exti​nguish the voices of mor​tal man from the sp​here I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful t​he final snuffing of the lie​s of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL I​S LOST the pon̷y he comes he c̶̮omes he comes the ich​or permeates all MY FACE MY FACE ᵒh god no NO NOO̼O​O NΘ stop the an​*̶͑̾̾​̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e n​ot rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚​N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

Source: https://stackoverflow.com/a/1732454

The post Sо уӧυ’гё sтϊll υsᎥng r̸e̴g͟e̸x̷ to parse H̶T̷M̵L appeared first on Be on the Right Side of Change.

You don’t need much for this tutorial:

1. Basic understanding of HTML, CSS, and JavaScript.
2. A modern web browser.

We’ll create a single HTML document that includes inline JavaScript and CSS. This approach ensures that you can easily copy, paste, and run the example without managing multiple files.

Execution: To run the Bitcoin price checker, copy the entire HTML code into a file with a .html extension. Open this file in any modern web browser to see the live Bitcoin price updating every few seconds.

You can see it live right here:

Here’s the full HTML document with embedded CSS and JavaScript:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Live Bitcoin Price Chart</title>
    <style>
        body {
            font-family: Arial, sans-serif;
        }
        #chartContainer {
            height: 300px;
            width: 100%;
            margin: 20px auto;
            border: 1px solid #ccc;
        }
    </style>
</head>
<body>
    <div id="chartContainer"></div>
    <script src="https://canvasjs.com/assets/script/canvasjs.min.js"></script>
    <script>
        let priceDataPoints = [];

        function updateChartData() {
            fetch('https://api.binance.com/api/v3/ticker/price?symbol=BTCUSDT')
                .then(response => response.json())
                .then(data => {
                    const btcPrice = parseFloat(data.price);
                    const currentTime = new Date();
                    priceDataPoints.push({ x: currentTime, y: btcPrice });
                    if (priceDataPoints.length > 20) {
                        priceDataPoints.shift();
                    }
                    updateChart();
                }).catch(error => console.error("Error fetching BTC price:", error));
        }

        function updateChart() {
            let chart = new CanvasJS.Chart("chartContainer", {
                title: {
                    text: "Live Bitcoin Price"
                },
                axisX: {
                    title: "Time",
                    valueFormatString: "HH:mm:ss",
                    labelAngle: -30
                },
                axisY: {
                    title: "Price (USD)",
                    includeZero: false,
                    prefix: "$"
                },
                data: [{
                    type: "line",
                    xValueType: "dateTime",
                    dataPoints: priceDataPoints
                }]
            });
            chart.render();
        }

        updateChartData(); // Initial call to populate the chart
        setInterval(updateChartData, 5000); // Update chart every 5 seconds
    </script>
</body>
</html>

The code works as follows:

HTML Structure: We have a div element where the chart will be displayed.

CSS Styling: Simple styles are applied to ensure the chart container is visually defined.

JavaScript for Live Data:

• Fetching Data: The JavaScript uses the fetch API to pull live Bitcoin prices from Binance’s public API endpoint.
• Updating the Chart: We use CanvasJS (a third-party library) to render the price chart. New data points are added every 5 seconds, and the oldest data points are removed to keep the chart clean and readable.
• CanvasJS Library: This is a powerful and easy-to-use chart library, which we include from a CDN. It simplifies drawing the live chart.

The post Creating a Live Bitcoin Price Chart Using Binance API (No API Key!) appeared first on Be on the Right Side of Change.

Trying to parse a site when web scraping or calling APIs with Python requests, but getting slapped with a 403 Forbidden error. Ouch! The basic code looks like this:

import requests
url = 'http://example.com/'
result = requests.get(url)
print(result.content.decode())

And the server is like, “Nope, you’re not coming in” , showing a 403 Forbidden error.

Turns out, servers can be picky. They might block requests that don’t seem to come from a web browser. For example, Elon Musk decided to block bots from scraping Twitter content to maintain data sovereignty in the AI race.

The trick to do it anyways? Add a User-Agent header to your request. It’s like wearing a disguise to look like a regular visitor.

Here’s how you do it:

import requests
url = 'http://worldagnetwork.com/'
headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
result = requests.get(url, headers=headers)
print(result.content.decode())

Still getting the cold shoulder with a 403 error? Time to beef up your disguise with more headers, like Referer.

You can find these headers in your browser’s developer tools (F12 → Network → Headers → Request Headers).

Here’s an enhanced header example:

headers = {
    'User-Agent': 'Your User Agent',
    'Referer': 'https://example.com'
}

Finding User-Agent: Too lazy to dig through the Network tab? Just type navigator.userAgent in the Chrome developer console.

Headers Missing in Network Tab?: Refresh the page, check any HTTP request, and scroll down to see the request headers.

What is the 403 Forbidden Error?

Have you been a nasty boy (or girl)?

The 403 Forbidden Error is like a strict doorkeeper saying you can’t enter. When you try to access a webpage, your browser sends a request. If the website’s server decides you shouldn’t access that page, it responds with a 403 Forbidden Error. It’s like the server saying, “I understand what you want, but I won’t let you in.” This happens for various reasons, like access control settings or website configurations.

Here’s a handy table of common HTTP error codes and their meanings.

These codes are part of the HTTP protocol, and they help developers understand what’s happening when their application interacts with web servers. They’re like a universal language for communicating the status of web requests.

Solving the Python Request Forbidden Error 403 When Web Scraping

When you’re web scraping with Python and hit a 403 error, it’s like knocking on a door and being turned away.

To solve this, change your approach. Use headers in your Python requests to mimic a real browser.

Here’s an example:

import requests

url = 'https://example.com'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)'}
response = requests.get(url, headers=headers)

This code tells the server, “Hey, I’m just like a regular browser, let me in!”

Python Requests Post Error 403

Facing a 403 error when sending a POST request in Python? This is like trying to submit a form on a website and being rejected.

To fix this, check if the website needs specific headers or cookies. Sometimes, including a ‘Referer’ header or a valid ‘User-Agent’ string helps:

import requests

url = 'https://example.com/post'
data = {'key': 'value'}
headers = {'User-Agent': 'Your User Agent', 'Referer': 'https://example.com'}
response = requests.post(url, data=data, headers=headers)

Here, you’re assuring the server that your request is legitimate.

Python Request Get 403 Error

When your Python GET request returns a 403 error, it’s like being denied entry when you ask for information. To bypass this, add headers that make your request look like it’s coming from a regular web browser:

import requests

url = 'https://example.com'
headers = {'User-Agent': 'Your User Agent'}
response = requests.get(url, headers=headers)

This code is your digital disguise to get past the server’s restrictions.

Python Requests Proxy Error 403

A 403 error when using a proxy in Python requests is like a bouncer blocking your disguised entry. Sometimes, servers block known proxies. Try using a different proxy or adding headers:

import requests

url = 'https://example.com'
proxies = {'http': 'http://10.10.1.10:3128'}
headers = {'User-Agent': 'Your User Agent'}
response = requests.get(url, proxies=proxies, headers=headers)

This approach is like changing your disguise and trying a different door.

Python urllib.request Error 403

Encountering a 403 error with urllib.request? It’s similar to using requests, but you’re using a different tool. Add a User-Agent in your request header:

import urllib.request

url = 'https://example.com'
req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0'})
response = urllib.request.urlopen(req)

This method tells the server that your request is coming from a common web browser.

403 Forbidden Delete Request

A 403 error on a DELETE request means you’re trying to remove something from the server, but it’s not allowing you. Check if you have the right permissions and if your request headers are correctly set. Sometimes, you also need an authentication token.

Getting the Authentication Token

An authentication token in the Python context, or in any programming context really, is a digital key that allows you to access certain resources or services. Imagine it like a special pass that lets you into a VIP area; without it, you’re not getting in.

When you’re coding in Python, especially when dealing with web APIs or web scraping, you often need to prove who you are to access certain data or functionalities. This is where the authentication token comes in. It’s a string (a series of characters) that verifies your identity to the server or service you’re trying to access.

Here’s a simple breakdown:

1. Obtaining the Token: First, you need to get this token. This usually happens after you log in or send a request with your credentials (like your username and password). The server then gives you a token as a response.
2. Using the Token: Once you have the token, you include it in the headers of your subsequent requests. This is like showing your pass every time you try to access something.
3. Server Verification: The server checks the token to ensure it’s valid and corresponds to a user with the right permissions. If everything checks out, you get access.

In Python, using an authentication token might look something like this:

import requests

url = 'https://example.com/api/data'
token = 'your-authentication-token-here'
headers = {'Authorization': f'Bearer {token}'}

response = requests.get(url, headers=headers)

In this example, the Authorization header is used to pass the token with each request, and the format Bearer {token} is a common way to present the token. The server, upon receiving this request, checks the token and then allows access to the data or functionality you requested.

403 Error Reasons

Common reasons for a 403 error include:

• Incorrect URL: Like dialing a wrong number.
• Access Control: The server’s way of saying, “You’re not on the guest list.”
• Firewall Settings: A digital guard blocking your path.
• Outdated Cache: Old information leading you astray.

Is Web Scraping Legal?

Web scraping walks a fine line between being super helpful and potentially troublesome. It’s legal if you respect the website’s terms of service and don’t overburden their server.

Think of it like fishing: do it responsibly, without depleting the fish population! Always check the website’s robots.txt file to understand their scraping rules.

Is Web Scraping Legal?

The post (Solved) Python Request Error 403 When Web Scraping appeared first on Be on the Right Side of Change.

At the end of this article, I’ll provide an interactive Google Colab link for the interactive version of this article. But let’s start with the video:

The Coding Challenge

Upwork is seen to be a good platform for potential freelance jobs.

But there can be some challenges in getting to the jobs quickly enough. Early applications are frequently the ones accepted.

The job search interface is also not very well suited to the filtering and listing of jobs you are looking for.

And…
Freelancers need to actively search the jobs page!

This tutorial and video look at a way to accelerate that search and filter the jobs on preferred criteria.

Learning Objectives

By the end of this tutorial, you will have:

• Defined the project requirements
• Explored aspects of Data Scraping
• Explored RSS feeds and XML
• Followed a potentially useful and repeatable workflow
• Built a useful tool
• Developed some useful Python skills

Approach

As freelancers, it can be helpful to approach every task as a formal project.
It is good practise and you never know when project might become something more valuable.

A client may want something similar or it may become a product you can sell.
So, it is a good discipline and will save time in the long run to approach such projects professionally.

A Useful Workflow

This pattern of development has proven helpful for me:

• Set the project requirements
• Follow a sound process for data scraping
  • Investigate the data source
  • Acquire the data
  • Extract the data you want
  • Clean the data
  • Filter the data
  • Output the data
  • Use the data and confirm the information is valid
• Document the project
• Deliver to the client

And we’ll follow this process now.

Requirements Setting

Just like we do for our clients, we should have specific requirements.

I use the MoSCow approach to setting my requirements.

This identifies the parts that the project we:

• Must Do
• Should Do
• Could Do
• Won’t Do

And sets out clearly what will be delivered and equally as important, what will not be delivered.

Our requirements:

MUST:

1. Provide data from Upwork relevant to the Freelancer
2. Present the information in a readable format

SHOULD:

1. Allow filtering and manipulation of the data as needed by the user
2. Allow for rapid refresh

COULD:

1. Run from the command line with arguments
2. Could be automated

WONT:

1. Have a graphical interface

We are limited on time. So we will focus on the data-scraping aspect of the task. And we will only complete the Must and Should requirements.

Getting the Data

Investigation

We can see here on the Upwork ‘search page’, entering a search term will give you a large number of potential tasks. But we only want some of these and preferably the latest. And we want them filtered to our needs. So we need to defer from what is presented here.

This symbol identifies the Really Simple Syndication (RSS) feed that we shall be using that feed for our data.

If we click the link and select RSS, a new page opens with the job feed structured in the Extensible Markup Language (XML).

We can also see that the feed format is XML.

This is similar to HTML and is a markup language that is readable by the computer and by people (people who can look past the tags and format).

Apparently this dense text is ‘person’ readable!

Some of it seems ok but most is hard to read.

Let’s make this more readable.

Time is money for the Freelancer.
So let’s copy this data and use a web tool, an XML formatter to explore the RSS XML data.

Here we can see that the XML forms a tree. With 10 indiviual elements, one for each job in this field.

If we look in to the first few items, we can see the information about each job.

Observations

It looks like the feed has 10 elements and each element has the Title, Link to the Job and a Description. The description appears to be an HTML script that contains some of the information that we need.

So let’s scrape that data next!

Acquisition

There are Python packages that we can use to scrape data from such XML feeds.

And it looks like we will be able to extract data from the ‘description’ field too. It appears to be a long string object and we have Python mehtods for strings.

For the RSS feed with will use ‘feedparser’

Note: For simple data-scraping tasks, I like to use a Jupyter notebook. The notebook is useful because it holds the data in memory so it can be explored while we change the code.

This means we don’t need to capture the feed many times.

There is no reason you can’t use VSCode or Pycharm or any other editor.

Again as a freelancer, time is money, so use the tools you are familiar with.

Looking at the feedparser documents

We can learn here that the data is acquired by and then parsed by

import feedparser
data = feedparser.parse(UPWORK_RSS_FEED_URL)

and then we can extract our 3 elements of data using:

import feedparser
data = feedparser.parse(UPWORK_RSS_FEED_URL)

item_title = data.entries[0].title
item_title
item_link = data.entries[0].link
item_link
description  = data.entries[0].description
description

Note: This data will change frequently as the RSS feed is updated with the latest jobs. This is why the RSS feed is so useful to us Finxters.

Extraction

We have already extracted the Title and the Link of the job just from the feedparser entries data.

item_title = data.entries[0].title
item_link = data.entries[0].link

Now we need to extract the individual elements of information from the ‘description‘ string.

Let’s take a closer look at one of the ‘description‘ strings.

description: We are looking for a skilled developer who can create a mobile application and web application for a fitness app. The main feature of the app will be the integration of AI technology to detect the user&#039;s body, diet, and workout plan. The successful candidate will be responsible for designing and developing the app, ensuring it is user-friendly and has a modern, sleek design. The app should be able to track user progress and provide personalized recommendations based on the user&#039;s inputs and body data. Key skills required for this project include: <br /><br />
- Mobile app development <br />
- Web app development <br />
- AI integration <br />
- UX/UI design <br />
- Data analysis and interpretation<br /><br /><b>Hourly Range</b>: $8.00-$10.00
<br /><b>Posted On</b>: December 02, 2023 17:57 UTC<br /><b>Category</b>: Mobile App Development<br /><b>Skills</b>:iOS, Android, Smartphone, Python, Mobile App Development
<br /><b>Skills</b>: iOS, Android, Smartphone, Python, Mobile App Development <br /><b>Country</b>: United States
<br /><a href="https://www.upwork.com/jobs/Fitness-App-Development-with-Functionality_%7E01494dc445d89c9f7f?source=rss">click to apply</a>

Here we see 14 lines of text with HTML markup and tag and characters such as ‘br‘ and &# 039 ;

Then we see a selection of headings inside HTML bold tags.

So the general theme for the description block is:

• description – HTML code of variable lengths and with some HTML character codes and tags
• Hourly Range – b_tags and some text
• Posted On – b_tags and some text
• Category – b_tags and some text
• Skills – b_tags and 1 or more skills with commas and spaces in between
• Skills – a repeated line of skills
• Country – b_tags and some text
• Link – a repeat of the link

Knowing this data structure, we can now use Python to extract the information we need.

Let’s write some code!

First we need to import some packages.

• feedparser for the RSS feed.
• pandas for our data storage and filtering
• ssl to bypass some ssl elements of the feed broadcast.

import feedparser
import pandas as pd
import ssl

Now we need a function to create and return an empty and prepared Dataframe in Pandas.

As we’ve discussed, we need to store each jobs:

• Title
• Link
• Description
• Posted on
• Category
• Skills List
• Price Type (Hourly Range or Budget
• Price of budget of max Hourly Rate
• Country the job originates in

def make_dataframe():

    jobs_df = pd.DataFrame(columns=[
        'Title', 
        'Link',
        'Description', 
        'Posted',
        'Category',
        'Skills',
        'Price Type', 
        'Price', 
        'Country' ])

    return jobs_df

We need a function that steps through the feed and extracts our information.

Firstly, we set up a list of blank data so that if there are gaps in the information, we still have data to place in the DataFrame. Failure to do this would raise an error.

Then we ‘Parse the Feed’

Title and Job link we can get directly from the feed entry.

But for the ‘description‘, we need to use the ‘string.split‘ method and split the string into a list of elements using the ‘bold‘ tag as the separator.

This gives us:

• description[0] is the first item in the list and is the main description field we just need to stip this of HTML tags Here we use the ‘clean_string‘ function.
• ‘Posted On’ and ‘Category’ also get cleaned with ‘clean_string‘.

Notice that we slice off only that part we need to send to be ‘cleaned’ eg clean_string(b_tag[15:])

• ‘Hourly Range’ / Budget’ get special treatment in the ‘clean_price’ function where we return a float for the money value and a string for ‘Budget’ or ‘Hourly Rate’
• ‘Skills’ needs to be stripped into a list (for searching) and also cleaned.
• ‘Country’ also needs some special treatment

Once cleaned the data is assigned to a dictionary and added to a DataFrame and added to the master DataFrame.

def get_data(entry):  # entry is a job item from the RSS feed

    # Some data ends up Null so set those values just in case
    item_posted = ''
    item_cat = ''
    item_price_type = ''
    item_price = 0.0
    item_skills = []
    item_country = ''

    # Set from parsing the feed
    item_title = entry.title
    item_link = entry.link
    description  = entry.description
    description = description.split('<b>')
    item_desc = clean_string(description[0])
    for b_tag in description[1:]:
        if "Hourly Range" in b_tag or "Budget" in b_tag:
            item_price_type, item_price = clean_price(b_tag)
        elif "Posted On" in b_tag:
            item_posted = clean_string(b_tag[15:])
        elif "Category" in b_tag:
            item_cat = clean_string(b_tag[14:])
        elif "Skills" in b_tag and not item_skills :
            item_skills = clean_skills(b_tag[11:])
        elif "Country" in b_tag:
            item_country = clean_country(b_tag[10:])




    # build the DataFrame and return it

    new_job = {
    'Title': item_title, 
    'Link': item_link, 
    'Description': item_desc,
    'Posted': item_posted,
    'Category': item_cat, 
    'Skills': item_skills, 
    'Price Type': item_price_type, 
    'Price': item_price, 
    'Country': item_country}

    new_job_df = pd.DataFrame([new_job])

    return new_job_df

The ‘clean_string‘ function uses the ‘replace‘ method and takes each substring that isn’t required and either removes it or replaces it with the correct value.

Note: This is not the most pythonic approach, but it has been written for clarity for beginners in mind. How would you make it more Pythonic?

def clean_string(string):

    string = string.replace('<br />','')
    string = string.replace('</b>','')
    string = string.replace('&nbsp;','')
    string= string.replace('&#039;','\'')
    string = string.replace('&rsquo;','\'')
    string = string.replace('&ldquo;','\"') 
    string = string.replace('&rdquo;','\"') 
    string = string.replace('quot;','\'')
    string = string.strip()

    return string

The ‘clean_price‘ function splits the identifier (‘Hourly Range’ or ‘Budget’) into a new string.

It then extracts the number (also a string) and returns it as a float along wth the identifier.

def clean_price(item_Bud_HR):

    price_split = item_Bud_HR.split(':')
    item_price_type = clean_string(price_split[0]) 
    item_price = price_split[1] # Get and clean the value
    item_price = item_price.replace('$','')
    item_price = item_price.replace('<br />','')
    item_price = item_price.strip()
    if '-' in item_price:
        item_price = item_price.split('-') # If the price is an 'Hourly Range' we split, returning the number on the right of  '-'  
        item_price = item_price[1]
    item_price = float(item_price)

    return item_price_type, item_price

The ‘clean_country‘ function splits the string on '\n'. It then takes the first element, cleans off the white space and returns the Country name.

def clean_country(item_country):

    item_country = item_country.split('\n')
    item_country = clean_string(item_country[0])
    item_country = item_country[1].strip()

    return item_country

The ‘clean_skills‘ function is a little more complex.

We create a new empty list, ‘item_skills_list‘.

We then clean the string by removing HTML tags.

We split the string on the ',' character and step through the list that is created, cleaning each string and then appending it to the list before it is returned.

def clean_skills(item_skills):

    item_skills_list =[]
    item_skills = item_skills.replace('<br />','')
    item_skills = item_skills.split(',')
    for skill in item_skills:
        item_skills_list.append(skill.strip())

    return item_skills_list

Once a new job DataFrame is created for each job, it is ‘concatenated’ to the master DataFrame for later filtering.

def join_dataframes(new_item_df,jobs_df):

    jobs_df = pd.concat([jobs_df, new_item_df], ignore_index=True)

    return jobs_df

The DataFrame jobs_df now holds all of the RSS feed jobs and their associated data. We can now filter it as required.

The ones I have presented here (commented out) offer examples for your own filters.

1. Strips out any duplicates based on the ‘Posted’ time.
2. Looks for budgets and hourly figures above 20.0 dollars
3. Looks for selected countries (United States and India)

What would you want to filter for?

def filter_output(jobs_df):


    # FILTER THE DATA USING Pandas

# 1. Strip out non unique values - Posted on is the pseudo-primary key
    # Uncomment if needed
    #jobs_df = jobs_df.drop_duplicates(subset=['Posted'], ignore_index=True) 

 #2. Only save for Price or budget greater than $10
    # Uncomment if needed
    #jobs_df = jobs_df[jobs_df['Price'] > 20] 

# 3. Only save for Specific Country 
    # selecting rows based on condition 
    # Uncomment below if needed
    #options = ['United States', 'India'] 
    #jobs_df = jobs_df[jobs_df['Country'].isin(options)]

    return jobs_df

Here we have the main() function that takes the Upwork RSS URL and feeds it to the function in turn.

# MAIN
def main():

# Why SSL
# Python is adding http verification in in the std library
# This bypasses the check for th moment
    if hasattr(ssl, '_create_unverified_context'):
        ssl._create_default_https_context = ssl._create_unverified_context


    url="https://www.upwork.com/ab/feed/jobs/rss?q=Python&sort=recency&paging=0%3B10&api_params=1&securityToken=6b9f07dc2632b4ac772d5daa37626af471b7d2526826c56a0c16aad6580245646f4e13804c72bd1ed3755f3bd552f5ba1d3f67a021987f714a1ff340ba7659dc&userUid=1215586676591329280&orgUid=1215586676603912193"

    #Get the Feed Data
    data = feedparser.parse(url)

    # Make the master dataframe
    jobs_df = make_dataframe()

    #Get the data for each item and add it to the DataFrame
    for entry in data.entries:
        new_job_df = get_data(entry)
        # join the new dataframe to the list
        jobs_df = join_dataframes(new_job_df, jobs_df)

Now we have the jobs from the RSS feed in a DataFrame, we can filter using the pandas methods.

    filter_output(jobs_df)

Once we have filtered the data to meet our needs, we use the pandas method to save the DataFrame to an Excel file.

It also prints out the top 3 entries to demonstrate the data has been captured.

    print(jobs_df.head(3))
    # Export to excel
    jobs_df.to_excel('jobs.xlsx', index=False)

                                          Title  \
0                AWS Python Consultant - Upwork   
1        1min Time Frame Forex Scalper - Upwork   
2  AI - driven crypto charting project - Upwork   

                                                Link  \
0  https://www.upwork.com/jobs/AWS-Python-Consult...   
1  https://www.upwork.com/jobs/1min-Time-Frame-Fo...   
2  https://www.upwork.com/jobs/driven-crypto-char...   

                                         Description  \
0  Fluent English speaking Python developer with ...   
1  If you scalp the forex market on the m1 time f...   
2  Scope of work\nThedevelopment of a crypto char...   

                        Posted            Category  \
0  December 02, 2023 20:13 UTC  DevOps Engineering   
1  December 02, 2023 20:13 UTC       Deep Learning   
2  December 02, 2023 20:03 UTC    Machine Learning   

                                              Skills    Price Type  Price  \
0  [Ubuntu, Amazon Web Services, Python, AWS Lamb...  Hourly Range   10.0   
1                                    [Forex Trading]  Hourly Range   40.0   
2  [Artificial Intelligence, Machine Learning, Bl...  Hourly Range   40.0   

          Country  
0  United Kingdom  
1  United Kingdom  
2         Nigeria  

That completes our exploration of the code.

I hope you have found some insight and value here.

Let us review.

Learning Objectives

This tutorial and video looked at how to read the RSS feed from Upwork, to accelerate your search and allow you to filter the jobs on your preferred criteria.

We have covered:

• Defining your project requirements
• Data Scraping
• RSS feeds and XML (briefly)
• A potentially useful workflow
• The building of a useful tool
• Some useful Python skills

Next Steps

This code is very flexible and so here are some options you might want to consider if you are extending its utility:

• You may want to run this code on a timer to give you frequent updates.
• You may also want to load the previous jobs scraped into the jobs_df DataFrame so that you can append new jobs.
• You may also want to have a ‘list of urls’ for different searches that you step through in order to cover lots of searches
• If you searches are very specific you might want to have the script email you when a job is posted.

What will you do?

Resources:

• https://jsonformatter.org/xml-formatter
• https://ascii.cl/htmlcodes.htm

You can also check out this guide on Google Colab using this link.

The post Improve Your Upwork Job Search with RSS and Data Scraping appeared first on Be on the Right Side of Change.

A good question often opens up new perspectives and new ways of thinking. But that’s not why I had to create a Question-Answering database.

Question-Answering (QA) databases play an important role in researching and developing large language models (LLMs). These databases have two primary applications:

• As training material
• To evaluate results

Traditionally, QA databases can be produced in two main ways:

• Manual compilation: experts or volunteers collect questions and answers, ensuring high quality and accuracy. (But slow and expensive)
• Crowdsourcing: using the public to collect questions and answers. (Quicker but less quality)

However, with today’s modern (Instruction Tuned) language models, it is possible to quickly generate high-quality databases at a low cost.

I could not find a suitable database in my language (Hungarian), so I had to create one.

In this article, I am creating a question-answer database based on Wikipedia pages determined by keywords. For this, I am using the OpenAI GPT-3.5 turbo model and the Wikipedia Python module.

How I Did It

After loading the necessary modules and setting the logging level (level=logging.DEBUG reports detailed information about the system’s operation, set this to "logging.INFO" if you don’t need this level of monitoring), we set up the Openai API.

Do not forget to provide your API key!

I statically encoded the instruction into the “messages” variable and everyone should adapt it to their own language. It is also worth trying different prompts for the best results.

I used this one: "Write 1 relevant question in Hungarian about the following text!"

# GPT 3.5 Turbo API Client
class GPT3ChatClient:
    def __init__(self, api_key='sk-YOUR_API_KEY'):
        self.api_key = api_key
        self.headers = {
            "Authorization": f"Bearer {self.api_key}",
            "Content-Type": "application/json"
        }
        self.endpoint = "https://api.openai.com/v1/chat/completions"

    def query(self, query_string, max_tokens=100, temperature=0.5, **kwargs):
        messages = [{"role": "user", "content": "Írj 1 releváns magyar nyelvű kérdést a következő szövegről! " + query_string}]     #You have to customize "content" to your language!
        payload = {
            "model": "gpt-3.5-turbo",
            "messages": messages,
            "max_tokens": max_tokens,
            "temperature": temperature,
            **kwargs
        }
        response = requests.post(self.endpoint, json=payload, headers=self.headers)
        response_data = response.json()
        return response_data['choices'][0]['message']['content'] if response_data.get('choices') else None

I have defined a function that transforms Hungarian accented characters, which will be necessary for automatic file name generation. (It’s completely optional!)

It uses the str.replace(char, new_char) function on a predefined dictionary of characters.

# Utility function for string replacement
def replace_accented_chars(input_str):
    replacements = {
        'á': 'a', 'é': 'e', 'í': 'i', 'ó': 'o', 'ö': 'o',
        'ő': 'o', 'ú': 'u', 'ü': 'u', 'ű': 'u', ' ': '_'
    }
    for accented_char, replacement in replacements.items():
        input_str = input_str.replace(accented_char, replacement)
    return input_str.lower()

The following function performs Wikipedia scraping with attention to the “results=2” parameter, which determines how many pages to search based on the keyword.

The second argument of the function (language="hu") is the language code. Set prefix to one of the two letter prefixes on the list of all Wikipedias.

I have integrated error handling into the function, and it uses “random.choice” to randomly select a page when a keyword would raise a DisambiguationError if the page is a disambiguation page.

The wikipedia.search() function has a “suggestion” argument, if True, it returns the results and suggestion (if any) in a tuple. I have not used this.

# Wikipedia scraping based on keywords
def scrape_wikipedia(keyword, language="hu"):
    wikipedia.set_lang(language)
    merged_text = []
    result = wikipedia.search(keyword, results=2, suggestion=False)

    for item in result:
        try:
            page = wikipedia.page(item, auto_suggest=False)
        except wikipedia.DisambiguationError as e:
            selected_option = random.choice(e.options)
            logging.debug(f"DisambiguationError: {selected_option}, {e}")
            page = wikipedia.page(selected_option)

        content = page.content
        merged_text.append(content)

    return merged_text

Defining some basic text cleaning:

In this section, I filter out non-alphanumeric (Hungarian) characters and replace the "\n" (newline) characters with spaces. To filter out short sentences, we only keep sentences longer than 5 words (Feel free to customize it!).

For an advanced LLM, it is advisable to provide a relatively longer context; this helps in generating better questions.

# Cleaning data
def clean_text(string_list):
    allowed_chars = re.compile('[a-zA-ZáÁéÉíÍóÓöÖőŐúÚüÜűŰ\s.:;,%?!0-9-]') #Allow only hungarian characters
    clean_list = [''.join(allowed_chars.findall(string.replace("\n", " "))) for string in string_list if len(string.split()) > 5]
    return clean_list

After that we initialize the GPT-3.5 ChatClient using an API key.

The API key grants access to the GPT service. We load a Spacy language processing model for the Hungarian language.

“hu_core_news_lg” is a pretrained model for the Hungarian language. Check if the Spacy model has a “sentencizer” factory, which is responsible for sentence splitting.

# Initialize GPT client and Spacy model
# api_key='sk-YOUR_API_KEY'
gpt_client = GPT3ChatClient(api_key) 
nlp = spacy.load("hu_core_news_lg")
assert nlp.has_factory("sentencizer")

I break down these pages into sentences using huSpacy’s CNN-based large model, but you can also get acceptable results using the following regex code (if no spacy model is available):

# Regular expression pattern for splitting text into sentences
def split_list_into_sentences(text_list):
    pattern = r'(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?|!)\s'
    sentences_list = []
    for text in text_list:
        sentences = re.split(pattern, text)
        sentences_list.extend(sentences)
    return sentences_list

Here, we set the keyword that we’ll use later to search for Wikipedia pages:

#Set the keyword for wikipedia search
keyword = "OpenAI" 

After that,

• We use the scrape_wikipedia function to fetch Wikipedia pages based on the provided keyword. The clean_text function is used to clean and preprocess the retrieved text.
• doc = nlp(str(cleaned_text)): We use Spacy to process the cleaned text, splitting it into individual sentences for further analysis.
• data = {...}: We create a data table that contains the processed sentences and queries to be asked to GPT-3.5. The queries are the sentences that we’ll ask ChatGPT about. This line creates a dictionary with two keys: 'Query' and 'Answer'. The values are lists generated using list comprehension.
  'Query': In this part of the code, a gpt_client object is used, and the query function is called for each sentence (sent.text) in the processed document (doc.sents).
  The result is a list containing questions from the LLM.
  'Answer': In this part of the code, each sentence from doc.sents, which was processed earlier, is simply inserted into a list. The result is a list containing all the sentences.
• df = pd.DataFrame(data): We create a Pandas DataFrame based on the data table created in the previous step.
• print(f"{len(df)} questions generated."): We print the number of generated questions.

In a corresponding prompt (You have to customize it for your language!), I generated questions for these answers, and then put the data into a DataFrame, from where it can be saved as a file and sent to, for example, human annotators or used for various projects.

Such a project could be, for example, an evaluation of a model’s “factual” knowledge on a given topic. By running through the generated questions, the existing “gold” answers can be compared with the answers given by the model.

Another, more fun use of the database could be, for example, to quickly set up quiz games.

For example, the DataFrame can be exported to an xlsx file to a given path:

# Write result to excel file
output_folder = "./results"
    if not os.path.exists(output_folder):    # create “results” folder if not exist
        os.makedirs(output_folder)
filename = replace_accented_chars(keyword)
df.to_excel(f"{output_folder}/{filename}.xlsx")

The result xlsx:

Conclusion

The created database contains approximately clean and relevant questions but, of course, it can be further refined (and is worth refining) with human effort, but this already involves much less effort than coming up with the questions from scratch.

Further improvement of the results can be achieved with additional prompting techniques, for example, by providing a system role to guide the language model toward the specific topic.

With the rise of large language models, many tasks are becoming faster, and considering that these models truly excel when working with texts, efficient code can be crafted for such work.

Related Blog: Prompt Engineering with Llama 2 (Full Course)

Full Code (Appendix)

Here is the full code:

#!/usr/bin/env python
# coding: utf-8

"""
Q&A Generator with GPT3.5-turbo from Wikipedia pages based on keywords.
"""

# Required libraries
import logging
import sys
import requests
import wikipedia
import random
import warnings
import re
import spacy
import huspacy
import pandas as pd
import os

# Logging configuration
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)

# GPT 3.5 Turbo API Client
class GPT3ChatClient:
    def __init__(self, api_key='sk-YOUR_API_KEY'):
        self.api_key = api_key
        self.headers = {
            "Authorization": f"Bearer {self.api_key}",
            "Content-Type": "application/json"
        }
        self.endpoint = "https://api.openai.com/v1/chat/completions"

    def query(self, query_string, max_tokens=100, temperature=0.5, **kwargs):
        messages = [{"role": "user", "content": "Írj 1 releváns magyar nyelvű kérdést a következő szövegről! " + query_string}]     #You have to customize "content" to your language
        payload = {
            "model": "gpt-3.5-turbo",
            "messages": messages,
            "max_tokens": max_tokens,
            "temperature": temperature,
            **kwargs
        }
        response = requests.post(self.endpoint, json=payload, headers=self.headers)
        response_data = response.json()
        return response_data['choices'][0]['message']['content'] if response_data.get('choices') else None

# Utility function for string replacement
def replace_accented_chars(input_str):
    replacements = {
        'á': 'a', 'é': 'e', 'í': 'i', 'ó': 'o', 'ö': 'o',
        'ő': 'o', 'ú': 'u', 'ü': 'u', 'ű': 'u', ' ': '_'
    }
    for accented_char, replacement in replacements.items():
        input_str = input_str.replace(accented_char, replacement)
    return input_str.lower()

# Wikipedia scraping based on keywords
def scrape_wikipedia(keyword, language="hu"):
    wikipedia.set_lang(language)
    merged_text = []
    result = wikipedia.search(keyword, results=2, suggestion=False)

    for item in result:
        try:
            page = wikipedia.page(item, auto_suggest=False)
        except wikipedia.DisambiguationError as e:
            selected_option = random.choice(e.options)
            logging.debug(f"DisambiguationError: {selected_option}, {e}")
            page = wikipedia.page(selected_option)

        content = page.content
        merged_text.append(content)

    return merged_text

# Cleaning data
def clean_text(string_list):
    allowed_chars = re.compile('[a-zA-ZáÁéÉíÍóÓöÖőŐúÚüÜűŰ\s.:;,%?!0-9-]')
    clean_list = [''.join(allowed_chars.findall(string.replace("\n", " "))) for string in string_list if len(string.split()) > 5]
    return clean_list

# Initialize GPT client and Spacy model
gpt_client = GPT3ChatClient(api_key='sk-')    #api_key='sk-YOUR_API_KEY'
nlp = spacy.load("hu_core_news_lg")
assert nlp.has_factory("sentencizer")

# Process keyword
keyword = "OpenAI"
cleaned_text = clean_text(scrape_wikipedia(keyword, language="hu"))    #Set the language code from here https://meta.wikimedia.org/wiki/List_of_Wikipedias

# Sentence splitting and DataFrame creation
doc = nlp(str(cleaned_text))
data = {
    'Query': [gpt_client.query(sent.text) for sent in doc.sents],
    'Answer': [sent.text for sent in doc.sents]
}
df = pd.DataFrame(data)
print(f"{len(df)} questions generated.")

# Write result to excel file
output_folder = "./results"
if not os.path.exists(output_folder):    # create “results” folder if not exist
    os.makedirs(output_folder)
filename = replace_accented_chars(keyword)
df.to_excel(f"{output_folder}/{filename}.xlsx")

# Optional file formats
#df.to_csv(f"{output_folder}/{filename}.csv")
#df.to_json(f"{output_folder}/{filename}.json")

The post How to Scrape Wikipedia Pages to Create a Question-Answer Database? appeared first on Be on the Right Side of Change.

Web scraping, a technique used to extract data from websites, has become an essential skill on Upwork — it’s one of the most sought-after skills on most freelancing platforms. Most beginners start with the Beautiful Soup and Requests modules in Python. While these tools are powerful, they’re not always sufficient for every site. Enter tools like Selenium, which, while powerful, can sometimes be overkill or inefficient.

So, where should one start? The answer is simple: Always check for an API first.

Why Start with APIs?

An Application Programming Interface (API) allows two software applications to communicate with each other. Many websites offer APIs to provide structured access to their data, making it easier and more efficient than scraping the web pages directly.

Benefits of using APIs:

• Efficiency: Extracting data from APIs is often faster and less resource-intensive than scraping web pages.
• Reliability: APIs are designed to be accessed programmatically, reducing the chances of breaking changes.
• Ethical considerations: Accessing data via an API is often more in line with a website’s terms of service than scraping their pages directly.

MindBodyOnline provides a dedicated API tailored for developers: MindBody API.

If you’re aiming to craft an app utilizing their dataset, this API is your ideal resource. It boasts a plethora of endpoints, enabling swift data retrieval and ensuring seamless interaction between your application and their servers.

But what if you aren’t creating an application and just need to scrape data once for research? MindBodyOnline also retrieves data for its website via an API. Javascript is used to request the data needed to populate their website. We can also make requests for this API.

How to check if a website is rendered with Javascript

The site we will be scraping is MindBodyOnline.

If a website is rendered with Javascript, we should check the network traffic and see if we can find a request that returns the data we see on the page. This can be done quickly with developer tools. With Chrome, you can bring up developer tools by clicking Ctl-Shift-I.

From here, we can turn off Javascript, then refresh the page and see if there are any changes. To turn off Javascript, first hit Ctl-Shift-P to bring up the command palette. Start typing Javascript to filter the options, then click “Disable javascript”.

Then refresh the page. As we can see, they use JavaScript for all the data.

Before we can continue, we need to turn JavaScript back on. Bring up the command palette again, filter for javascript, and click “Enable Javascript”. Then refresh the page again.

Check the JavaScript Requests

Select the Network tab in developer tools.

Make sure Fetch/XHR and Preserve log are selected. Next, we can click the circle with the line through it to clear the output. Then perform a search to see what requests were performed.

We can then check each item in the output to see if it returns useful information.

We are primarily interested in the response to the request. We are looking for XML data that looks like the data shown on the page. In this case, it is the locations request that contains the data we seek.

We can also see that there is a payload required. When we make our requests, we must provide this payload in the request body. There are three items of interest here. The latitude and longitude allow us to control the city we are pulling data for, and we also need to provide a page number.

MindBody uses pagination, so a relatively small amount of data is pulled with each request. A large city like New York can have over a hundred pages.

We go to the headers tab to copy the request URL.

Using Insomnia to Generate Request Headers

From here, we can use a tool to help us with the request syntax.

Insomnia is a powerful open-source API client tool for testing and debugging APIs. It provides a user-friendly interface to send requests to web services and view responses. With Insomnia, you can define various request types, from simple HTTP GET requests to complex JSON, GraphQL, or even multipart file uploads. You can download the insomnia desktop app here.

Using Insomnia is quite simple. Just paste in the API URL and click Send.

We can check the preview tab to make sure it returns the data we want:

This is where it gets good. If we click the dropdown on the send button, one of the options is “generate client code”. How convenient! Just click Python as the language and use the Requests module and you can click “Copy to Clipboard” and you’re off to the races.

A Simple Scrapy Spider

The code can be found on Github. I will walk through the code below, starting with the imports.

import scrapy
import json
import pandas as pd
from scrapy.crawler import CrawlerProcess
import os

Scrapy is a good option because it can handle multiple requests at the same time with asynchronous processing. Scapy has a lot of bells and whistles and a fair bit of a learning curve, but it’s also possible to avoid a lot of the extra complexity. The goal here was to place all the code in one simple script.

First, we have to create a spider class. The class is pretty large so I’ll display it in chunks.

class MindbodySpider(scrapy.Spider):
    name = 'mindbody_spider'

    custom_settings = {
        'CONCURRENT_REQUESTS': 5,
        'DOWNLOAD_DELAY': 3.2,
    }

Our class inherits from one of the Scrapy Spider classes with scrapy.Spider being the simplest. In the custom settings, with CONCURRENT_REQUESTS set to 5, scrapy will be processing six requests at a time, starting a new one as soon as one finishes.

We use a DOWNLOAD_DELAY so we don’t bombard the website with too many requests at once.

Next, we need a starting template for the payload

starting_payload = '''{
                          "sort":"-_score,distance",
                          "page":{"size":50,"number":<<num>>},
                          "filter":{"categories":"any",
                                    "latitude":<<lat>>,
                                    "longitude":<<lon>>,
                                    "categoryTypes":"any"}
                       }'''

Next, we have the headers that Insomnia so helpfully provided for us.

headers = {
        "cookie": "__cf_bm=zdIhLHXKd2OAveBChKORUMdydUFVzC2Ma51sQxv.UJ0-1694646164-0-Abmbwcj2wNw%2FpityY4DWRWy%2FftBkjTO0vQ3tZ0gwU0P5bsTqcasf2XZlBwL%2BUaevGaH%2BTDzZOJPBXbWYwgsXkJc%3D",
        "authority": "prod-mkt-gateway.mindbody.io",
        "accept": "application/vnd.api+json",
        "accept-language": "en-US,en;q=0.9",
        "content-type": "application/json",
        "origin": "https://www.mindbodyonline.com",
        "sec-ch-ua": "^\^Not/A",
        "sec-ch-ua-mobile": "?0",
        "sec-ch-ua-platform": "^\^Windows^^",
        "sec-fetch-dest": "empty",
        "sec-fetch-mode": "cors",
        "sec-fetch-site": "cross-site",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36",
        "x-mb-app-build": "2023-08-02T13:33:44.200Z",
        "x-mb-app-name": "mindbody.io",
        "x-mb-app-version": "e5d1fad6",
        "x-mb-user-session-id": "oeu1688920580338r0.2065068094427127"
    }

Then a very simple init method

def __init__(self):
        scrapy.Spider.__init__(self)
        self.city_count = 0

The start_requests method loops through each city. This is the main loop that creates the first request for each city.

def start_requests(self):
        cities = pd.read_csv('uscities.csv')

        for idx, city in cities[].iterrows():
            lat, lon = city.lat, city.lng
            self.logger.info(f"{city.city}, {city.state_id} started")

            # Start with the first page for each city
            payload = self.starting_payload.replace('<<pg>>', '1').replace('<<lat>>', str(lat)).replace('<<lon>>', str(lon))

            yield scrapy.Request(
                url="https://prod-mkt-gateway.mindbody.io/v1/search/locations",
                method="GET",
                body=payload,
                headers=self.headers,
                meta={'city_name': city.city, 'page_num': 1, 'lat': lat, 'lon': lon, 'state': city.state_id},
                callback=self.parse
            )

The code is pretty simple. We create a DataFrame from a CSV file with city information and then loop through it with the iterrows method. We create the payload for the request using the template and the lat/long values from the DataFrame. The page is set to 1 each time. We will handle additional pages later.

Finally, we yield a scrapy.Request object. We use yield instead of return so we can handle multiple requests concurrently. The body is our modified payload, and we use the same header for each request.

What do we do with the response returned from the request? As soon as the response is returned it is fed into the parse method thanks to the callback parameter:

callback=self.parse

The meta parameter gives us a way to pass information to the callback function. We need the page num, lat, lon values for the next request.  City_name and state are used for screen outputs.

The list of cities was downloaded off the web. Many different options will work, as long as they contain latitude and longitude values.

Parsing the Response

The parse method is a little long, but not too complicated.

Getting the data and saving it is very easy. We just convert response.text to a DataFrame and save it to a CSV file. If the file already exists, we will append the data and not include a header. Otherwise, we create a new CSV file and include a header.

def parse(self, response):
        data = json.loads(response.text)
        gyms_df = pd.json_normalize(data['data'])

        # Save the dataframe to a CSV
        city_name = response.meta['city_name']
        state = response.meta['state']
        fname = f'{city_name} {state}.csv'.replace(' ', '_')
        csv_path = f'./data/cities2/{fname}'

        # Check if file exists to determine the write mode
        write_mode = 'a' if os.path.exists(csv_path) else 'w'

        gyms_df.to_csv(csv_path, 
                       mode=write_mode, 
                       index=False, 
                       header=(not os.path.exists(csv_path)))         

Handling Pagination

To move on to the next page, we need to create another Scrapy Request. For the payload we use the same latitude and longitude and increment the page number by 1.

        # Check if there's another page and if so, initiate the request
        next_page_num = response.meta['page_num'] + 1
        if next_page_num <= 150:  # Optional: upper limit
            lat, lon = response.meta['lat'], response.meta['lon']  # Assuming you store lat and lon in meta too

            payload = self.starting_payload.replace('<<pg>>', '1').replace('<<lat>>', str(lat)).replace('<<lon>>', str(lon))

Make the Request for the Next Page

To finish the parse method, all we have to do is make another request with the new payload.

yield scrapy.Request(
                url="https://prod-mkt-gateway.mindbody.io/v1/search/locations",
                method="GET",
                body=payload,
                headers=self.headers,
                meta={'city_name': response.meta['city_name'], 
                      'page_num': next_page_num, 
                      'lat': lat, 
                      'lon': lon,
                      'state': state},
                callback=self.parse
            )

        self.city_count += 1
        print(response.meta['city_name'], f'complete ({self.city_count})')
        self.logger.info(f"""{response.meta['city_name']}, 
                           {response.meta['state']} is complete""")

How the Pagination Loop Terminates

What happens if there are 100 pages for the current city and the code sends a request with page_num = 101?

The request will not return anything, so the callback function won’t get called and the recursive loop for that city will stop.

Then the start_requests loop will move on to the next city.

It’s alive! Setting Our Little Spider Loose

To get our creepy critter crawling, we create a CrawlerProcess. Then tell it to crawl. Then tell it to start. On your mark, get set, CRAWL!

process = CrawlerProcess()
process.crawl(MindbodySpider)
process.start()

Results

I was able to scrape data for 16,000 cities in about half a week. I think I averaged about 100 cities an hour. The larger cities had over a hundred pages but there were thousands upon thousands of cities with 5-10 pages.

What about the data? It’s fairly extensive and could be very useful.

Pretty good information related to services offered, location, amenities, total ratings etc. Looking at the rest of the columns:

Conclusion

Uncovering the API proved invaluable. It eliminated the need to craft path selectors for individual data elements, significantly streamlining the process. Moreover, it spared me from devising a Scrapy workaround for the JavaScript-rendered page. Investing time in learning Scrapy was a sound decision, given its superior speed compared to other methods I explored.

Looking ahead, the logical progression is to integrate the data into platforms like Jupyter Notebook, Power BI, or Tableau. Furthermore, storing the data in a database seems apt, especially considering the apparent one-to-many relationships observed in each city, like categories and subcategories.

If you want to become a master web scraper, feel free to check out our academy course with downloadable PDF certificate to showcase your skills to future employers or freelancing clients:

Academy: Web Scraping with BeautifulSoup

The post How I Scraped Data From Over 16,000 Gyms from MindBodyOnline.com appeared first on Be on the Right Side of Change.

The search_google(query, num_results) function performs a search using the Google Custom Search JSON API, with the search query and the desired number of results (num_results) as parameters.

Upon a successful API request (HTTP Status Code 200), the function extracts and prints titles, URLs, and snippets of the search results.

Subsequently, it employs the get_texts_from_urls function to retrieve the text content from each URL, which internally uses get_text_from_url to send a GET request to each URL, parse the HTML using BeautifulSoup, and extract the text. The extracted texts are then printed along with their respective lengths.

The API key (google_search_api) and Custom Search Engine ID (google_search_cx) are needed to authenticate and specify the search engine, respectively. Both are free though.

Note: The API key and Custom Search Engine ID should be obtained from Google Cloud Console and Programmable Search Engine, respectively, and should be kept confidential to prevent unauthorized usage. Both are free for up to 100 requests per hour or so.

Otherwise, you can just copy and paste the following code into your Jupyter notebook (or Google Colab):

import requests
from bs4 import BeautifulSoup


keyword = 'get stock price in excel cell'
google_search_api = '...' # https://developers.google.com/custom-search/v1/introduction
google_search_cx = '...' # https://programmablesearchengine.google.com/controlpanel/create


def search_google(query, num_results=10):
    '''Returns a list of strings with the text of the first n google results'''

    # Google Custom Search JSON API URL
    api_url = "https://www.googleapis.com/customsearch/v1"
    
    # Parameters for the API request
    params = {
        'key': google_search_api,  # Your API key
        'cx': google_search_cx,  # Your Custom Search Engine ID
        'q': query,  # The search query
        'num': num_results  # The number of results to retrieve
    }
    
    # Make the API request
    response = requests.get(api_url, params=params)
    
    # Check if the request was successful (HTTP Status Code 200)
    if response.status_code == 200:
        # Parse the JSON response
        results = response.json()
        
        titles = [item['title'] for index, item in enumerate(results.get('items', []))]
        print(titles)

        urls = [item['link'] for index, item in enumerate(results.get('items', []))]
        print(urls)

        snippets = [item['snippet'] for index, item in enumerate(results.get('items', []))]
        print(snippets)

        # Get the Text of all URLs
        texts = get_texts_from_urls(urls)
        print(texts)

        for text in texts:
            print(len(text))

        

    else:
        print("Failed to retrieve results:", response.status_code, response.text)


def get_text_from_url(url):
    try:
        # Send a GET request to the URL
        response = requests.get(url)
        
        # Check if the request was successful (HTTP Status Code 200)
        if response.status_code == 200:
            # Parse the HTML content of the page with BeautifulSoup
            soup = BeautifulSoup(response.text, 'html.parser')
            
            # Extract text from the parsed HTML
            return soup.get_text(separator="\n", strip=True)
        else:
            print(f"Failed to retrieve content from {url}. Status code: {response.status_code}")
            return None
    except Exception as e:
        print(f"An error occurred while fetching {url}: {str(e)}")
        return None


def get_texts_from_urls(urls):
    texts = []
    for url in urls:
        text = get_text_from_url(url)
        if text:
            texts.append(text)
    return texts



search_google(keyword, 5)

If you don’t use Google Colab Notebook to run the code (e.g., in your own .py script), you may need to install BeautifulSoup and Requests — check out the linked articles for a quick tutorial on accomplishing this.

For the provided keyword 'get stock price in excel cell', the output of this code was as follows:

['Get a stock quote - Microsoft Support', 'The formula that returns real-time stock prices', 'Excel data types: Stocks and geography - Microsoft Support', 'Add Real-Time Stock Prices And Metrics To Excel', 'Get stock price (latest close) - Excel formula | Exceljet']
['https://support.microsoft.com/en-us/office/get-a-stock-quote-e5af3212-e024-4d4c-bea0-623cf07fbc54', 'https://www.morningbrew.com/money-scoop/stories/2021/08/24/formula-returns-realtime-stock-prices', 'https://support.microsoft.com/en-us/office/excel-data-types-stocks-and-geography-61a33056-9935-484f-8ac8-f1a89e210877', 'https://www.thespreadsheetguru.com/add-real-time-stock-prices-and-metrics-to-excel/', 'https://exceljet.net/formulas/get-stock-price-latest-close']
["Type some text in cells. · Then select the cells. · Although it's not required, we recommend creating an Excel table. · With the cells still selected, go to the\xa0...", 'Aug 24, 2021 ... In Excel: Yep, Excel can return stock prices, too. Enter each ticker in its own cell > Highlight the cells > Select “Data” > “Stocks” > Tap\xa0...', '... price are getting extracted from the Geography data type in column A. Type some text in cells. If you want stock information, type a ticker symbol, company\xa0...', 'Jul 28, 2019 ... If you click the Convert to 

...

problem with this page\nHelp us improve Exceljet\nYou must have JavaScript enabled to use this form.\nName\nEmail\nProblem\n*\nYour email address is private and not shared.\nExceljet\nQuick, clean, and to the point\nResources\nTraining\nVideos\nFunctions\nFormulas\nShortcuts\nArticles\nAbout Us\nAbout\nTestimonials\nTopics\nContact\nNewsletter Sign-up\nWork faster in Excel.\nJoin more than 100,000\npeople who get weekly tips from us.\nEmail\nHP\n© 2012-2023 Exceljet.\nTerms of use\nVisit our Twitter page']
6189
9885
8960

Thanks for reading the article and feel free to join our 100% free email academy by downloading our Python cheat sheets:

The post Get First N Google Search Results in Python (Official Way) appeared first on Be on the Right Side of Change.

JavaScript Regex

In the online tool above, I used a JavaScript regular expression:

var emailRegex = /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g; 

The variable emailRegex holds a regular expression, which is a pattern used to match character combinations in strings, specifically designed here to identify email addresses.

In the pattern /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g, [a-zA-Z0-9._%+-]+ matches the username part of the email, allowing for a combination of uppercase and lowercase letters, numbers, and several special characters.

The @ symbol is a literal match, ensuring the presence of the at-symbol in the email.

[a-zA-Z0-9.-]+ matches the domain name, allowing letters, numbers, periods, and hyphens. \. ensures a literal period is present, and [a-zA-Z]{2,} matches the top-level domain (like com, org, net), ensuring it is composed of at least two letters.

The g at the end of the pattern indicates a global search, meaning it will match all instances of the pattern in the provided string, not just the first one it encounters.

This regular expression provides a general pattern for identifying email addresses within a text, although it’s worth noting that validating email addresses using a regular expression can get exceedingly complex due to the wide variety of valid formats specified by the standard.

Regex to Match Email Addresses in Different Programming Languages

Fortunately, regular expressions (regex) tend to be quite consistent across different programming languages and environments, as many languages implement regular expressions in a way that adheres to a standard (often, the Perl 5 regular expression syntax). However, there can be slight variations or additional features in some languages.

Below is a general regex pattern for matching email addresses, along with examples of how it might be implemented in various programming languages:

General Regex Pattern:

[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}

1. JavaScript:

var emailRegex = /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/g;

2. Python (using re module):

import re
email_regex = re.compile(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')

You can also check out this Finxter article to learn more about scraping websites to extract email addresses. However, make sure you have the right to do so!

3. Java:

Pattern emailRegex = Pattern.compile("[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}");

4. C# (.NET):

Regex emailRegex = new Regex(@"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}");

5. PHP (using preg_match):

$emailRegex = '/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/';

6. Ruby:

email_regex = /[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/

7. Swift:

let emailRegex = "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}"

8. Kotlin:

val emailRegex = "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}".toRegex()

9. Perl:

my $email_regex = qr/[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}/;

10. R:

email_regex <- "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}"

Note that the above regex pattern is quite basic and will cover many, but not all, valid email address formats. Email validation can get quite complex if you want to cover all valid email formats as specified by the standards (RFC 5322, RFC 6531). The above examples should work for many common use cases, but be sure to test thoroughly with your own data to ensure no valid email addresses are being missed.

Have fun, thanks for reading, and keep learning!

Feel free to join our free email newsletter to stay tuned about exciting tech news while learning coding and AI tools, as well as prompt engineering:

The post Regex to Extract All Email Addresses from a String ✅ (Tutorial + Online Tool) appeared first on Be on the Right Side of Change.

Quick Answer

When web scraping with BeautifulSoup in Python, you may encounter relative URLs (e.g., /page2.html) instead of absolute URLs (e.g., http://example.com/page2.html). To convert relative URLs to absolute URLs, you can use the urljoin() function from the urllib.parse module.

Below is an example of how to extract absolute URLs from the a tags on a webpage using BeautifulSoup and urljoin:

from bs4 import BeautifulSoup
import requests
from urllib.parse import urljoin

# URL of the webpage you want to scrape
url = 'http://example.com'

# Send an HTTP request to the URL
response = requests.get(url)
response.raise_for_status()  # Raise an error for bad responses

# Parse the webpage content
soup = BeautifulSoup(response.text, 'html.parser')

# Find all the 'a' tags on the webpage
for a_tag in soup.find_all('a'):
    # Get the href attribute from the 'a' tag
    href = a_tag.get('href')

    # Use urljoin to convert the relative URL to an absolute URL
    absolute_url = urljoin(url, href)

    # Print the absolute URL
    print(absolute_url)

In this example:

• url is the URL of the webpage you want to scrape.
• response is the HTTP response obtained by sending an HTTP GET request to the URL.
• soup is a BeautifulSoup object that contains the parsed HTML content of the webpage.
• soup.find_all('a') finds all the a tags on the webpage.
• a_tag.get('href') gets the href attribute from an a tag, which is the relative URL.
• urljoin(url, href) converts the relative URL to an absolute URL by joining it with the base URL.
• absolute_url is the absolute URL, which is printed to the console.

Now that you have a quick overview let’s dive into the specific problem more deeply and discuss various methods to solve this easily and effectively.

Problem Formulation

Problem: How do you extract all the absolute URLs from an HTML page?

Example: Consider the following webpage which has numerous links:

Now, when you try to scrape the links as highlighted above, you find that only the relative links/paths are extracted instead of the entire absolute path. Let us have a look at the code given below, which demonstrates what happens when you try to extract the 'href' elements normally.

from bs4 import BeautifulSoup
import urllib.request
from urllib.parse import urljoin
import requests

web_url = 'https://sayonshubham.github.io/'
headers = {"User-Agent": "Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) "
                         "Chrome/31.0.1650.0 Safari/537.36"}
# get() Request
response = requests.get(web_url, headers=headers)
# Store the webpage contents
webpage = response.content
# Check Status Code (Optional)
# print(response.status_code)
# Create a BeautifulSoup object out of the webpage content
soup = BeautifulSoup(webpage, "html.parser")
for i in soup.find_all('nav'):
    for url in i.find_all('a'):
        print(url['href'])

Output:

/
/about
/blog
/finxter
/

The above output is not what you desired. You wanted to extract the absolute paths as shown below:

https://sayonshubham.github.io/
https://sayonshubham.github.io/about
https://sayonshubham.github.io/blog
https://sayonshubham.github.io/finxter
https://sayonshubham.github.io/

Without further delay, let us go ahead and try to extract the absolute paths instead of the relative paths.

Method 1: Using urllib.parse.urljoin()

The easiest solution to our problem is to use the urllib.parse.urljoin() method.

According to the Python documentation: urllib.parse.urljoin() is used to construct a full/absolute URL by combining the “base URL” with another URL. The advantage of using the urljoin() is that it properly resolves the relative path, whether BASE_URL is the domain of the URL, or the absolute URL of the webpage.

from urllib.parse import urljoin

URL_1 = 'http://www.example.com'
URL_2 = 'http://www.example.com/something/index.html'

print(urljoin(URL_1, '/demo'))
print(urljoin(URL_2, '/demo'))

Output:

http://www.example.com/demo
http://www.example.com/demo

Now that we have an idea about urljoin, let us have a look at the following code which successfully resolves our problem and helps us to extract the complete/absolute paths from the HTML page.

Solution:

from bs4 import BeautifulSoup
import urllib.request
from urllib.parse import urljoin
import requests

web_url = 'https://sayonshubham.github.io/'
headers = {"User-Agent": "Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) "
                         "Chrome/31.0.1650.0 Safari/537.36"}
# get() Request
response = requests.get(web_url, headers=headers)
# Store the webpage contents
webpage = response.content
# Check Status Code (Optional)
# print(response.status_code)
# Create a BeautifulSoup object out of the webpage content
soup = BeautifulSoup(webpage, "html.parser")
for i in soup.find_all('nav'):
    for url in i.find_all('a'):
        print(urljoin(web_url, url.get('href')))

Output:

https://sayonshubham.github.io/
https://sayonshubham.github.io/about
https://sayonshubham.github.io/blog
https://sayonshubham.github.io/finxter
https://sayonshubham.github.io/

Method 2: Concatenate The Base URL And Relative URL Manually

Another workaround to our problem is to concatenate the base part of the URL and the relative URLs manually, just like two ordinary strings. The problem, in this case, is that manually adding the strings might lead to “one-off” errors — try to spot the extra front slash characters / below:

URL_1 = 'http://www.example.com/'
print(URL_1+'/demo')

# Output --> http://www.example.com//demo

Therefore to ensure proper concatenation, you have to modify your code accordingly such that any extra character that might lead to errors is removed. Let us have a look at the following code that helps us to concatenate the base and the relative paths without the presence of any extra forward slash.

Solution:

from bs4 import BeautifulSoup
import urllib.request
from urllib.parse import urljoin
import requests

web_url = 'https://sayonshubham.github.io/'
headers = {"User-Agent": "Mozilla/5.0 (CrKey armv7l 1.5.16041) AppleWebKit/537.36 (KHTML, like Gecko) "
                         "Chrome/31.0.1650.0 Safari/537.36"}
# get() Request
response = requests.get(web_url, headers=headers)
# Store the webpage contents
webpage = response.content
# Check Status Code (Optional)
# print(response.status_code)
# Create a BeautifulSoup object out of the webpage content
soup = BeautifulSoup(webpage, "html.parser")
for i in soup.find_all('nav'):
    for url in i.find_all('a'):
        # extract the href string
        x = url['href']
        # remove the extra forward-slash if present
        if x[0] == '/':       
            print(web_url + x[1:])
        else:
            print(web_url+x)

Output:

https://sayonshubham.github.io/
https://sayonshubham.github.io/about
https://sayonshubham.github.io/blog
https://sayonshubham.github.io/finxter
https://sayonshubham.github.io/

Caution: This is not the recommended way of extracting the absolute path from a given HTML page. In situations when you have an automated script that needs to resolve a URL but at the time of writing the script you don’t know what website your script is visiting, in that case, this method won’t serve your purpose, and your go-to method would be to use urlljoin. Nevertheless, this method deserves to be mentioned because, in our case, it successfully serves the purpose and helps us to extract the absolute URLs.

Conclusion

In this article, we learned how to extract the absolute links from a given HTML page using BeautifulSoup. If you want to master the concepts of Pythons BeautifulSoup library and dive deep into the concepts along with examples and video lessons, please have a look at the following link and follow the articles one by one wherein you will find every aspect of BeautifulSoup explained in great details.

Recommended: Web Scraping With BeautifulSoup In Python

With that, we come to the end of this tutorial! Please stay tuned and subscribe for more interesting content in the future.

The post Python BS4 – How to Scrape Absolute URL Instead of Relative Path appeared first on Be on the Right Side of Change.