Giggle GPT Joke Search Engine – Implementing Pagination (5/6)

by

Hi and welcome back to part 5, where we will be implementing pagination and testing to see how our page holds up when there really are a large number of jokes saved.

This is part 1 of the following series:

🎓 Full 6-video course with downloadable PDF certificates: Giggle – Creating a Joke Search Engine from Scratch with ChatGPT

As always, make sure your virtual environment is running. I’m going to assume you’re familiar with the command by now.

Now if you’ve been playing around with the jokes for a little bit as I have, you might have noticed the list becoming very very long. We will address this issue using pagination, or splitting up the huge list of items into multiple dynamically generated pages.

Django makes it fairly easy for us to create a paginator.

What a paginator does is basically just divide a huge list of results into fixed-size chunks so we can treat them as if they were different pages. Think of a forum thread with a huge number of replies where it gets divided over multiple pages.

Go to your views.py file in the giggle folder and first import the built-in Django Paginator by adding this import near the top of the file.

from django.core.paginator import Paginator

The most standard place is to add it with the other Django imports before you declare your .apis and .models import like this:

from django.shortcuts import render, redirect, get_object_or_404
from django.core.paginator import Paginator
from .apis import get_giggle_result
from .models import Joke

With that import added, scroll all the way down to your giggle_history view function. Let’s upgrade our giggle_history view function like this, adding the paginator object to get started:

def giggle_history(request):
    jokes = Joke.objects.all().order_by('-created_at')
    paginator = Paginator(jokes, 8)
    return render(request, './giggle/giggle_history.html', context={'jokes': jokes})

We still get all the jokes first, that has not changed.

Next, we called the Paginator function Django provided for us and passed in our jokes stating we would paginate them in chunks of 8, which is the second argument.

We catch the returning paginator object in the variable we named paginator. More code is needed to make this work but we will need to take a moment to discuss URL query parameters first.

URL query parameters

In a little bit, we will be creating links to get to the different pages in the paginator. They will look like this.

www.example.com/history?page=5

You have probably seen this syntax before on many of the pages you visit on a daily basis. The question mark starts a query and then we simply define a key-value pair of page equals 5.

This is a way to send variables with a GET request instead of a POST request, so there is no need to enter data into a form, the data is simply part of the URL path.

The advantage of using these kinds of URL embedded variables is that if a user decided there is an amazing joke on the third page and they share the URL with you ending on '?page=3' you will click on the link and be taken to the same page the other user had shared.

If your website used private post data to access the right page this information would not be stored in the URL and users could not share the exact page they were looking at with others.

We will be using this type of link in a moment but first, we will get back to our giggle_history view function to be able to handle incoming links of this type and extract the key-value pairs.

Now we know how the URLs will have the ?page=x data appended to it, we can finish our giggle_history view function like this:

def giggle_history(request):
    jokes = Joke.objects.all().order_by('-created_at')
    paginator = Paginator(jokes, 8)
    page_no = request.GET.get('page', 1)
    curr_page = paginator.get_page(page_no)
    return render(request, './giggle/giggle_history.html', context={'jokes': curr_page})

We create a variable called page_no.

The request.GET specifies the method, which is GET, or just the normal loading of a page as any page you open on the internet.

The second .get() with lowercase letters is the .get method which is used to retrieve the data we passed to the GET with capital letters request using the URL appended data, the '?page=x'.

The key name of the data we want to get from the URL is ‘page’, and the second argument provides a fallback default value of 1 if no page number is specified.

We then create a variable curr_page (current_page) and to set its value we call the paginator’s get_page() method, passing in the page number we want.

This will basically just return a subset of the complete jokes data to us. For example, page 1 would yield jokes 1 through 8 to us and page 2 would yield jokes 9 through 16, assuming you chose a page size of 8 like me.

Then finally make sure you edited the return statement as above. The context will no longer take all the jokes but only the curr_page subset of the jokes. Do not change the key’s name of ‘jokes‘, as this would affect your template which uses this variable name.

Now run your Django development server (python manage.py runserver) and go to the history page. You will now see a maximum of no more than 8 jokes on the page.

Now try visiting the following address (make sure to change the port/address to match your own dev-server as needed):

http://127.0.0.1:8000/history?page=2

You will now see the second page of 8 jokes, or possibly less depending on how many jokes you have. If you only have 13 you will have a first page of 8 and a second page of 5 jokes, but if you have many jokes, you will have many possible pages.

Paginator Navigation

Next, we will need a way for the user to comfortably navigate these pages.

We will need to have links to the previous and the next page dynamically based on which page we’re on and how many pages are available in total.

Say we’re on page 1, we will not have a previous page and a link to page number 2 as next page, but if we’re on page 3 we need a previous page link to page 2 and a next page link to page 4. Luckily Django is going to help us out here as well.

Let’s go back to our giggle_history.html template and we will insert a new section after the closing of the final ‘div‘ and before the closing of the ‘main‘ tag.

Give yourself some space to work here by pressing enter a couple of times, for all the following code blocks will go inside this space.

                ...
            {% endif %}
        </div>





    </main>

{% endblock %}

Remember we passed in only a subset of the total collection of jokes, a subset generated by the paginator. The paginator has sent several other helpers along in the context for us besides just the subset of jokes that will make creating our page links a lot easier.

The joke object does not just contain just our subset of jokes but also these paginator helpers just described.

First, add the following code in the gap.

{% if jokes.has_previous %} 
    <a href="?page={{ jokes.previous_page_number }}">previous</a>
{% endif %}

In the first line, we call jokes.has_previous.

This is one of the helpers of the paginator that I was talking about. If there is a previous page in the paginator available, it will return true, else false.

So only if there is a previous jokes page will the link on the next line be rendered onto the page. The link is to 'this page' with the URL query we learned about appended to it, '?page=x'.

In this case, the x is the number provided by another paginator helper in the form of jokes.previous_page_number, which provides exactly what the name suggests.

Now add this one directly below:

{% if jokes.has_next %} 
    <a href="?page={{ jokes.next_page_number }}">next</a>
{% endif %}

As you can see it follows the exact same logic and the paginator once again provides us with similar helpers in the form of .has_next and .next_page_number.

Go ahead and reload your history page after saving. You are now able to click back and forth depending on how many pages and jokes you have. If there are at least three pages you will have both a previous and a next link on your number 2 page.

If you don’t have a ton of joke pages to test with don’t worry about it, we will create a bunch of test data to stress test our paginator a bit later on in the tutorial.

It would be nice for the user to know which page they’re currently looking at without having to look at the URL.

In the giggle_history where we have been working, in between the .has_previous and .has_next conditional blocks you just added, simply add:

{{ jokes.number }}

The .number is again provided by our friend the lovely paginator. Now reload the page and see the current page number in between our pagination links.

Django Filters

Before we can take this one step further, we need to take a small detour and talk about filters in the Django template engine. They essentially allow you to run some template syntax value inside the {{ curly braces }} through a filter before outputting to the HTML.

To employ these we use the | pipe symbol.

This idea is quite similar to Linux bash scripting if you’re familiar with this. It’s easiest to just think of it as piping the output of whatever is on the left side of the pipe into the function on the right of the pipe.

Let’s take a look at how this works. Say we want to display not the current page number but the current page number + 2, just for the sake of trying this out. So if the current page is page 2 we want to display a number 4 instead of 2.

Go to the {{ jokes.number }} line you last wrote, and change it into:

{{ jokes.number | add:'2' }}

We first take the value of the current page number using jokes.number.

Whatever this value is then gets piped from the left side of the pipe to the right side into a filter with the name 'add:'. This filter takes an argument, the number you want to add to the filter’s input, behind the colon and inside brackets.

Now if you reload the page we will not see the current page number but the number of two pages ahead. Displaying the wrong page number is a terrible idea, of course, so go ahead and delete this wrong code, but we now have enough knowledge to take our paginator a step further!

Taking It Up a Notch

First, wrap the whole giggle_history paginator inside of an HMTL div element, and then delete the middle single line in between the 'jokes.has_previous' and 'jokes.has_next' code blocks.

Your paginator will end up like this:

<div class="paginator">
    {% if jokes.has_previous %} 
        <a href="?page={{ jokes.previous_page_number }}">previous</a>
    {% endif %}

    <<<<<This is where the new code will go in the next step>>>>>

    {% if jokes.has_next %} 
        <a href="?page={{ jokes.next_page_number }}">next</a>
    {% endif %}
</div>

Now insert the following into the gap in the middle between the has_previous and has_next blocks.

<div class="pg-numbers">
    {% for num in jokes.paginator.page_range %}
        {% if jokes.number == num %}
            {{ num }}
        {% elif num > jokes.number|add:'-5' and num < jokes.number|add:'5' %}
            <a href="?page={{ num }}">{{ num }}</a>
        {% endif %}
    {% endfor %}
</div>

So let’s go over this. The div requires no explanation.

jokes.paginator.page_range is just a list of all existing page numbers. We start a for loop, looping over each number in this list of existing page numbers and testing each number against the two conditionals inside the loop.

If the currently on-screen page’s number (jokes.number) is the same as the number variable in the current iteration of the loop, this is the current page number, just go ahead and render the number to the HTML as is.

Else if the number variable in the current loop iteration is between the range of current-page -5 and current-page +5, we print a link to this particular page number.

Remember we get the current page number by calling jokes.number, and then use the add: filter with either a negative or positive number to add or subtract.

Note, since the list of possible page numbers we loop over will run from lowest number to higher, all these things being rendered to the screen will be rendered in order from lowest to highest as each loop runs. Go ahead and check out your site on the development server to test it out.

Our pagination is really starting to take shape now! Let’s fix up the styling a bit. Go to your styles.css file in the static folder and add to the bottom.

.paginator {
    margin-top: 2em;
    display: flex;
    justify-content: space-between;
}

We select the div which we gave the class of paginator in our HTML template and apply flexbox using 'display: flex' which then allows us to specify we want equal spacing between all children of this particular div element.

If you check out your site on the development server you will now see a ‘previous’ link on the left, your numbered pagination in the middle, and a ‘next’ link on the right-hand side.

The only problem we have is that our first and last pages do not have a ‘previous’ and ‘next’ link, and therefore the numbers get squashed all the way to the side.

To fix this, find the following block back in your giggle_history.html file again:

{% if jokes.has_previous %} 
    <a href="?page={{ jokes.previous_page_number }}">previous</a>
{% endif %}

And change it to:

{% if jokes.has_previous %} 
    <a href="?page={{ jokes.previous_page_number }}">previous</a>
{% else %}
    <span class="disabled">previous</span>
{% endif %}

Then find the has_next code block:

{% if jokes.has_next %} 
    <a href="?page={{ jokes.next_page_number }}">next</a>
{% endif %}

And make the same changes there:

{% if jokes.has_next %} 
    <a href="?page={{ jokes.next_page_number }}">next</a>
{% else %}
    <span class="disabled">next</span>
{% endif %}

Now both will display a disabled textual version of the link when no previous or next page is available. Go back to your styles.css file in the static folder and add the following styles:

.disabled {
    color: #AAA;
}

.pg-numbers {
    letter-spacing: 0.1em;
}

One gives a color to the disabled span we just created and one spaces the page numbers in the middle slightly apart. Go ahead and load your page in the development server and you should now have a pretty nice pagination system in place!

In case you got confused anywhere, I will provide the entire giggle_history.html code in a single block here before we continue on:

{% extends "./base.html" %}  

{% block content %} 

    <main>

        <div class="history">
            {% if jokes %}
                {% for joke in jokes %} 
                <div class="joke">
                    <h3>{{ joke.query }}</h3>
                    <a href="{% url 'delete_joke' pk=joke.id %}">
                        <span class="material-icons trash-icon">delete</span>
                    </a>
                    <p>{{ joke.response }}</p>
                </div>
                {% endfor %}
            {% endif %}
        </div>

        <div class="paginator">
            {% if jokes.has_previous %} 
                <a href="?page={{ jokes.previous_page_number }}">previous</a>
            {% else %}
                <span class="disabled">previous</span>
            {% endif %}

            <div class="pg-numbers">
                {% for num in jokes.paginator.page_range %}
                    {% if jokes.number == num %}
                        {{ num }}
                    {% elif num > jokes.number|add:'-5' and num < jokes.number|add:'5' %}
                        <a href="?page={{ num }}">{{ num }}</a>
                    {% endif %}
                {% endfor %}
            </div>

            {% if jokes.has_next %} 
                <a href="?page={{ jokes.next_page_number }}">next</a>
            {% else %}
                <span class="disabled">next</span>
            {% endif %}
        </div>

    </main>

{% endblock %}

Generating Testing Data

So now that our paginator works, let’s really give it a proper test. I barely have three pages of jokes right now, which means we’ll need to generate some data. Now let’s create a new file in our giggle folder called utils.py. This file will live in the same folder as your apis.py, views.py, etc.

Open this file and start with an import as the first line.

from .models import Joke

Then below, create a loop that will save the same joke n number of times over so we can really test our paginator to the limit.

def test_data_generator(amount):
    for i in range(0, amount):
        new_joke_object = Joke(
            query = "zombie watermelons",
            response = "Why did the zombie watermelon break up with his girlfriend? Because he thought she was a little too rotten!"
        )
        new_joke_object.save()

This simple function takes an amount as an argument and then loops over a range from zero to the specified amount, in each loop creating a new joke object, manually specifying the query and the response, and saving it to the database.

This bypasses our API call and just saves hardcoded data to the database. This type of function can be helpful sometimes to quickly stress test your own code with more or certain data.

Now go to our views.py file. Below the 'from .models import Joke' line import the function you just created:

from .utils import test_data_generator

Then, without changing anything about your giggle_search view function, just add a call to your test_data_generator function as the very first line inside the function.

def giggle_search(request):
    test_data_generator(100)
    context = {}
    .....keep all the view function code unchanged below...

Specify however many jokes you want to generate in the brackets. Now load your main giggle search page once and only once!

Then immediately remove the test data generator call again 'test_data_generator(100)' from your view function again so the code looks just like it did before.

We could have of course created a separate route and all to host this test data generator but since we only wanted to run it once we just used the main giggle search page view function to run it.

If you now check out your website’s history page, you will see a lot of zombie watermelon jokes!

Go ahead and really give the paginator a good test. Always test your code and really make sure that it works properly as you expected. We have a good selection of page options on the left and right depending on how many previous and next pages are available.

Fixing an Old Bug and Using Django Messages

Before we get too excited about our project, we have to go back in time a little bit and fix an old bug still on our to-do list. Remember that we are limited to 3 API calls per minute as we use the free API. Go ahead and open the apis.py file we wrote back then.

We decided to catch the RateLimitError to stop the server from crashing. We made the function return an error code of 1 to indicate the call was unsuccessful.

In our views.py giggle_search function, we call the get_giggle_result function we imported from our apis.py, but we never test if it returns this error code.

The result is that if we make several requests in quick succession, the 4th joke will be saved as the query with the response being '1', as that is what our function returned to views.py.

Let’s search for 4 topics in very quick succession:

Huge tower
Why did the huge tower break up with its girlfriend? Because she was always taking it for granite!

Tennis balls
Why do tennis players never get served in restaurants? Because they always bring their own balls!

Hippies
Why did the hipster burn his tongue? He drank his coffee before it was cool.

Bananas
1

Oops! We haven’t actually handled the return code 1, with unforeseen results.

To aid in our communication with the user, we’re going to look at one last Django feature (though there are many more!) called messages. It will allow us to show an informational 1-time-only alert to the user in a non-obnoxious manner.

So let’s finish our error handling here. Go back to your views.py file.

First of all, remove the 'from .utils import test_data_generator' import up top, as it is no longer needed. Now add a new import up top:

from django.contrib import messages

Now scroll down to our giggle_search function and find the following line inside, where we call the API:

response = get_giggle_result(query)

Below it, insert our condition to test for the error response code '1'.

response = get_giggle_result(query)
if response == 1:
    messages.error(request, 'You have hit the rate limit of 3 requests per minute, please try again in a moment.')
    return render(request, './giggle/giggle_search.html', context)

If the response equals '1', we will add an error message to the messages module by calling messages.error and passing in the request object and the message we want to display.

Then we return, so no other code gets executed like saving this to the database, and render the base giggle search page again with an empty context (as there is no valid joke to display).

Now, if there is an error, Django will send an error using the messages module. Where will this error message go? It will be passed via the context to the templates we are rendering. This means that in our templates we need to conditionally render error messages if any exist.

Let’s go to our base.html template file and locate the div with the class of “container”.

<div class="container">
    {% include './logo.html' %}  
    {% block content %}{% endblock %} 
</div>

Before the logo and content blocks, insert a conditional testing for the presence of any messages.

<div class="container">
    {% if messages %}
        {% for message in messages %}
            <div role="alert">
                {{ message }}
            </div>
        {% endfor %}
    {% endif %}

    {% include './logo.html' %}  
    {% block content %}{% endblock %} 
</div>

The code should look fairly familiar by now. If there are messages, for each message in the list of messages, create a div and put the message inside. Now if you go ahead and enter a query and then refresh the page a couple of times, suddenly our error message will appear!

The placement is very awkward right now so let’s fix the 'if messages' conditional statement.

{% if messages %}
    {% for message in messages %}
        <div class="alert alert-danger alert-dismissible fade show message" role="alert">
            {{ message }}
            <button type="button" class="btn-close mx-2 my-2 p-0" data-bs-dismiss="alert" aria-label="close"></button>
        </div>
    {% endfor %}
{% endif %}

We will use Bootstrap for some quick styling. Remember we installed Bootstrap in part 1.

It provides us with utility classes to quickly style this alert. The alert class gives it basic alert styling, alert-danger makes it a red errory message and alert-dismissible will allow us to add a close message button.

Fade and show add a basic CSS transition and finally, we added a class named message which we will use ourselves in our own CSS file to finish the styling later.

The btn-close will allow us to use this as a close button. The mx-2 class deals with margin on the x-axis, setting it to 2, my is a margin y-axis class and p is used for padding and set to 0 in this case.

This is the way that CSS libraries can save you some time with these utility classes so we don’t have to write yet another CSS selector for just one tiny button. If you ever need a specific style like this message box you can just copy the needed classes from the Bootstrap docs, so don’t worry too much about the details of the HTML itself.

Now if you test your error message it looks a lot better already, and has a working close button!

The close button works because of Bootstrap, which also includes some JavaScript, which is the reason we didn’t have to code the close button ourselves.

Now go to your styles.css in the static folder to finish the message box styling using the extra 'message' class we added for our own use.

Add the following selector to your CSS:

.message {
    position: absolute;
    top: 0;
    left: 0;
    width: 100%;
    text-align: center;
    border-radius: 0;
}

We use position absolute to move our message out of the normal document flow. We now have absolute positioning powers to set the distance from the top and the left of the page to 0 and the width to 100%. We set the text-align to center and remove the border-radius bootstrap added in.

Now we have a nice message that shows up and informs us if we exceed the rate limit, check it out!

To finish this off let’s add another selector to our styles.css file to help with the spacing of our jokes on the history page to make it look a little bit better.

.joke h3 {
    margin-bottom: 0.15em;
    margin-top: 0.7em;
}

Before we wrap up this part, we need to do a little bit of cleanup.

Let’s go to our views.py file, and see if you can find any code that seems to no longer serve any purpose. It’s inside the giggle_search function.

previous_jokes = Joke.objects.all().order_by('-created_at')[:2]
context['old_jokes'] = previous_jokes

These previous_jokes sections were for displaying the previous 2 jokes on the search page, but we got rid of this. Both the previous_jokes / old_jokes parts can be removed from the GET and the POST conditional blocks.

As this leaves the GET conditional block empty, it itself can also be removed. After getting rid of this stale code your giggle_search function will look like this:

def giggle_search(request):
    context = {}
    if request.method == 'POST':
        query = request.POST.get('query')
        if len(query) > 50:
            return redirect('giggle_search')
        response = get_giggle_result(query)
        if response == 1:
            messages.error(request, 'You have hit the rate limit of 3 requests per minute, please try again in a moment.')
            return render(request, './giggle/giggle_search.html', context)
        context['query'] = query.capitalize()
        context['response'] = response
        new_joke_object = Joke(query=query, response=response)
        new_joke_object.save() 

    return render(request, './giggle/giggle_search.html', context)

Sometimes you just need to have a look and see if all your code is actually still relevant and optimal.

That’s it for part 5! Feel free to play around with any of the styling and layout if you want to.

We’ve kept it fairly basic as it is not the main focus of this tutorial. Next up is part 6, where we’ll explore ChatGPT and the basic issues with it on our website, and also using ChatGPT to increase our own coding productivity.