Imagine you’ve spent hours working on a Python project, creating complex data structures, and suddenly, you need to save your progress, close your script, and transfer your data. What can you do? Well, that’s where Python’s pickle module comes to the rescue!

Pickle allows you to easily persist your Python objects by converting them into a binary format, also known as “pickling.” When you need to restore your data, you can simply “unpickle” it, bringing your objects back to life. The best part? The pickle module works with a wide range of Python types, including custom-defined objects, making it incredibly versatile for all of your serialization needs.

So, get ready to dive into the world of Python pickling, where you’ll discover how to save and resume your work effortlessly, ensuring that your hard-earned data remains intact across sessions and even across different systems.

Serialization in Python

When working with Python, you might encounter situations where you need to persist your objects for future use. This is where the pickle module comes in handy. With pickle, you can easily serialize and deserialize Python objects to store and retrieve them as needed.

Serialization is the process of converting an object into a sequence of bytes, which can be later deserialized back into the original object. This can be especially useful when you need to save the state of an object or transfer data between different programs.

In Python, the pickle module supports the serialization and deserialization of a wide range of objects, including lists, dictionaries, and custom class instances. To serialize an object, you simply call the pickle.dumps() method, like so:

import pickle
my_list = [1, 2, 3]
serialized_data = pickle.dumps(my_list)

To deserialize the data, you can use the pickle.loads() method:

deserialized_data = pickle.loads(serialized_data)

You’ll find that deserialized_data is now an exact replica of the original list, my_list.

Keep in mind that while pickle is a powerful tool for storing and loading Python objects, it has certain limitations. It can’t serialize objects containing certain types of data, such as opened files or lambda functions. Nonetheless, it’s an indispensable module for working with Python objects persistently. Happy pickling!

What is the Python Pickle Module?

The Python pickle module is a powerful tool to serialize and deserialize objects in Python. Unlike the JSON module, which serializes objects into a human-readable format, pickle uses a binary format for serialization, making it faster and compatible with more Python types right out of the box, including custom-defined objects. (source)

Think of pickling as a process that converts a complex Python object hierarchy into a byte stream. Once serialized, you can easily write the byte stream to a file, send it across a network, or store it in a database. Unpickling, on the other hand, is the inverse operation that transforms a byte stream back into the original object hierarchy. (source)

What makes pickle truly remarkable is its ability to serialize and deserialize a wide range of Python objects, such as lists, dictionaries, and your own custom classes. To effectively use the pickle module, keep in mind the following methods:

• pickle.dump() – Serialize an object to a file-like object
• pickle.load() – Deserialize an object from a file-like object
• pickle.dumps() – Serialize an object to a byte stream
• pickle.loads() – Deserialize an object from a byte stream

While using pickle, remember to exercise caution when deserializing untrusted data, as it might be a security risk. Now that you know the basics of the Python pickle module, you’re ready to explore its capabilities in serializing and deserializing objects.

How to Install Pickle

Good news! If you’re using Python 3, the pickle module comes pre-installed with your Python distribution, so there’s no need to perform additional installations. Simply use the import pickle command to get started with serializing and deserializing your Python objects!

If, for some reason, the standard import doesn’t work, you can try installing the pickle-mixin package by running the following command in your terminal or command prompt:

pip install pickle-mixin

Keep in mind that the pickle-mixin package is intended to provide compatibility with older Python versions and might not be necessary for Python 3.9 or later.

That’s it! You’re now ready to start using the pickle module to save and restore your Python objects. If you’re using older Python versions (3.5-3.7), you can consider the pickle5 package, which backports the features and APIs added in the pickle module in Python 3.8.3, including the PEP 574 additions. To install, simply run:

pip install pickle5

With pickle installed and ready to go, you can now easily persist your Python objects and work smoothly with complex data structures. Good luck, and happy pickling!

Basic Operations of Pickle

In this section, you will learn about the basic operations of the Python pickle module, which allows you to persist objects in Python effectively. We will cover pickling and unpickling objects, two essential processes for object serialization and deserialization.

Pickling Objects

Pickling is the process of converting a Python object into a byte stream, suitable for storing on disk or sending over a network. To pickle an object, you can use the pickle.dump() function. Here is an example:

import pickle

data = {"key": "value"}  # An example dictionary object to pickle
filename = "data.pkl"

with open(filename, "wb") as file:
    pickle.dump(data, file)

In this example, you create a dictionary named data, then save it to a file named data.pkl using the pickle.dump() function. The "wb" mode in open() indicates you’re writing to the file in binary mode, which is necessary for pickling.

Unpickling Objects

Unpickling is the process of converting a byte stream back into a Python object. To unpickle an object, you can use the pickle.load() function. Here is an example:

import pickle

filename = "data.pkl"

with open(filename, "rb") as file:
    loaded_data = pickle.load(file)

print(loaded_data)  # This will output: {"key": "value"}

In this example, you open the "data.pkl" file that contains the pickled object in binary reading mode ("rb"). You then load the object using the pickle.load() function and store it in the variable loaded_data.

Now you know the basics of pickling and unpickling objects in Python using the pickle module! You can use these techniques to serialize and deserialize your objects, enabling easy persistence and storage.

Python Pickle Examples

To get started with pickling in Python, make sure to import the pickle module:

import pickle

Now, let’s dive into some examples!

Example 1: Pickling a Dictionary

First, create a dictionary and store it in a file:

data = {"one": 1, "two": 2, "three": 3}
with open("data.pickle", "wb") as file:
    pickle.dump(data, file)

In this example, you use pickle.dump() to write the serialized object (the dictionary) to a file. Make sure to open the file in write-binary ('wb') mode [DigitalOcean].

Example 2: Unpickling a Dictionary

Now that you have pickled your dictionary, you can retrieve it from the file:

with open("data.pickle", "rb") as file:
    loaded_data = pickle.load(file)
print(loaded_data)

Here, you use pickle.load() to read and deserialize the object stored in the file. Note that the file should be opened in read-binary ('rb') mode.

Example 3: Pickling and Unpickling a List

You can also pickle and unpickle more complex data structures like lists:

fruits = ["apple", "banana", "orange"]
with open("fruits.pickle", "wb") as file:
    pickle.dump(fruits, file)

with open("fruits.pickle", "rb") as file:
    loaded_fruits = pickle.load(file)
print(loaded_fruits)

This time, you’re pickling a list of fruits. The pickling and unpickling process is the same as in previous examples. Just remember to use the appropriate binary mode when opening the file .

Picklable and Unpicklable Types

When working with the Python pickle module, it’s essential to understand which types of objects can be pickled and which cannot. Let’s dive into the details of picklable and unpicklable types in Python.

Picklable types are Python objects that can be serialized into byte streams using the pickle module. Most built-in Python types, including integers, floats, strings, tuples, lists, and dictionaries, are picklable. Moreover, custom-defined objects and instances of classes can also be pickled as long as they meet certain requirements, such as not including unpicklable attributes.

Here are some examples of commonly used picklable types:

• Numbers (int, float, complex)
• Strings
• Tuples, lists, and dictionaries (containing only picklable items)
• Functions and classes defined at the top level of a module
• Instances of classes, excluding those that cannot be pickled

On the other hand, unpicklable types include objects that cannot be serialized using the pickle module. These might be objects that have external dependencies or simply don’t support pickling.

Some examples of unpicklable types are:

• Open file objects
• Network sockets
• Database connections
• Lambda functions
• Objects containing unpicklable attributes

When working with the pickle module, ensure your data consists of picklable types to avoid errors and bugs. If you come across an unpicklable type, consider using alternative methods or libraries for serialization.

Remember that pickling is a powerful tool for persisting objects in Python, so understanding picklable and unpicklable types is crucial to utilize this module effectively.

Handling Errors in Pickle

When using the Python pickle module, you may encounter errors. Don’t worry! Understanding common errors and their causes can be helpful in resolving these issues. Below are some common pickle errors and tips on how you might handle them:

• PicklingError: This is the base class for all pickle-related exceptions. It usually occurs when you are trying to serialize an object that is not pickleable.
• UnpicklingError: An exception raised when pickle data is invalid or corrupted.

Now, let’s take a look at some steps you can follow to handle pickle errors effectively:

1. Verify that objects you’re pickling are serializable. Python can serialize many built-in datatypes, but not all types are pickleable. For example, you can’t pickle functions, classes, or lambda expressions. If you need to pickle these, consider using helper methods like __getstate__ and __setstate__ to customize serialization.
2. Double-check that the imported modules containing classes you need to pickle are available. Pickle stores class names, not the classes themselves, so it’s important that your script can access the module when unpickling. This is a common source of errors when working with pickled classes.
3. Be mindful of security concerns with pickle. The module is not secure and untrusted data can execute arbitrary code during unpickling! To handle these concerns, avoid unpickling data from untrusted sources, and validate your data before pickling if possible. More on this can be found in the official pickle documentation.

By keeping these tips in mind, you can handle errors in pickle smoothly and make the most of its powerful object serialization capabilities. Remember, practice makes perfect!

How to Compress Pickled Objects

Pickling in Python is a way to serialize objects, which makes it possible to store them in a file or transmit them over a network. However, pickled objects can sometimes take up a significant amount of space. In this section, you’ll learn how to compress pickled objects to save both storage and bandwidth.

The Python pickle module offers several ways to persist objects but doesn’t provide built-in compression. To compress pickled objects, you can use the bz2, gzip, or zlib module. Let’s go through the steps to compress and decompress pickled objects using the bz2 module:

1. Pickle the object: First, you need to serialize the object using the pickle.dumps() function, which returns a pickled byte string.
2. Compress the pickled byte string: Once you have your pickled byte string, use the bz2.compress() function to compress it.
3. Save or transmit the compressed data: You can store the compressed data in a file, send it to another user, or even upload it to a cloud storage service like S3.
4. Decompress the data: When you want to use the pickled object again, decompress the data using the bz2.decompress() function.
5. Unpickle the object: Finally, use the pickle.loads() function to deserialize the compressed data back into the original object.

Here’s a code example that demonstrates the process:

import pickle
import bz2

# Serialize your object
your_object = {"key": "value"}
pickled_object = pickle.dumps(your_object)

# Compress the pickled object
compressed_pickled_object = bz2.compress(pickled_object)

# Decompress the pickled object
decompressed_pickled_object = bz2.decompress(compressed_pickled_object)

# Deserialize the object
unpickled_object = pickle.loads(decompressed_pickled_object)

Using compression can drastically reduce the size of your pickled objects, making them easier to store and transmit. As you can see, compressing and decompressing pickled objects in Python is a simple and straightforward process. Now that you know how to do this, you can start optimizing your serialized data with ease. Happy pickling!

Security Considerations

When using the Python pickle module to persist objects, it’s crucial to keep security in mind. The pickle module is not secure and can potentially execute arbitrary code during the unpickling process . Therefore, it’s essential to only unpickle data you trust 100%.

Keep in mind that pickle is not intended to be secure against erroneous or maliciously constructed data. You should be cautious when unpickling data, especially if it comes from an untrusted or unauthenticated source.

• Be mindful of which sources you obtain pickled data from, and verify their authenticity.
• Always use regular validation when storing and retrieving data using pickle to protect against potential attacks.
• Prevent exposing sensitive information by limiting access to stored pickled data to only authorized users.

Following these security considerations will help you to maintain a safe environment when using the Python pickle module, while still enjoying its benefits for object persistence. Remember, always prioritize security when working with data in Python .

Pickle Alternatives

Though the Python pickle module is useful for object serialization, it’s essential to know its alternatives to make an informed decision. Here, we discuss some popular options that provide similar functionality and can be used depending on your specific needs.

JSON: For a more human-readable serialization format, consider using the built-in JSON module. JSON is a widely adopted and versatile format that works well for encoding simple data structures such as lists and dictionaries. However, it might not support custom-defined Python objects directly. Take a look at the JSON documentation for more information.

marshal: Another built-in Python module, marshal, can also be used for object serialization. However, this module’s primary use is for writing .pyc files, and its documentation explicitly states that pickle or JSON should be preferred for general persistence tasks. Keep in mind that marshal is not safe to use with untrusted data as it could lead to security risks.

MessagePack: If you require a more compact and efficient binary serialization format, consider MessagePack. This library is an excellent choice when working with large data sets and supporting multiple languages. It boasts faster serialization and deserialization speeds while maintaining a small message size.

Protobuf: Developed by Google, Protocol Buffers (Protobuf) is another popular alternative for data serialization. This robust and efficient binary format excels in performance, offers backward and forward compatibility, and provides language-agnostic support. It’s especially suited for large-scale distributed systems and communication between different services.

In summary, you have various alternatives to choose from when it comes to object serialization in Python. Each option has its strengths and trade-offs, depending on what you prioritize in your application. Take some time to explore these options and decide which one best suits your requirements. Happy coding!

Use Cases and Applications

When working with Python, you may come across situations where you need to persist objects for reasons such as data retrieval or further analysis. The Python pickle module can assist you in this process. Here are some common use cases and applications where pickle can prove to be beneficial:

• Data Persistence: Using pickle, you can store your program’s state data to disk, allowing it to carry on where it left off when restarted (source). This is particularly useful when you want to maintain information between application sessions or when dealing with large datasets that cannot be processed in a single run.
• Communications: Pickle can assist in sending Python data over TCP connections in multi-core or distributed systems (source). Through serialization and deserialization, pickle allows for the smooth transfer of complex Python objects between different processes or networked devices.
• Database Storage: Objects in Python can be stored in databases with the help of pickle (source). Pickle’s ability to handle a wide range of Python types makes it suitable for persisting instances of your custom-defined classes or other complex structures in databases, enabling you to later access and utilize them with ease.

While pickle has several advantages, be cautious that the serialized objects are in binary format and not human readable. Though this makes pickle faster than JSON, it also means that extra care has to be taken when working with serialized objects, especially when it comes to security and integrity of the stored data.

Python Pickle List

When working with Python, you may need to store and retrieve complex data structures, such as lists. The Python pickle module offers an easy way to achieve this. In this section, you’ll learn how to pickle and unpickle lists using Python’s pickle module.

To begin with, let’s import the pickle module:

import pickle

Now, suppose you have a list and wish to save it to disk. You can use the pickle.dump() function along with a file object, like this:

my_list = [1, 2, 3, 4, 5]
with open('my_list.pkl', 'wb') as file:
    pickle.dump(my_list, file)

Here, the 'wb' flag indicates that the file is opened for writing in binary mode. Your list (my_list) will be pickled and stored in the file called 'my_list.pkl'.

To unpickle and load the list back into your Python program, use the pickle.load() function:

with open('my_list.pkl', 'rb') as file:
    loaded_list = pickle.load(file)
print(loaded_list)

This time, the 'rb' flag is used to read the file in binary mode. Your pickled list will be loaded into the loaded_list variable, and you can use it just like any other Python list.

Remember, when using the Python pickle module to persist objects, such as lists, always use "wb" and "rb" flags for writing and reading files in binary mode, respectively. This will ensure that your data remains safe and intact.

Python Pickle Files

When working with Python’s pickle module, you may need a simple yet effective way to persist objects. The process of converting complex data structures, such as Python objects, into a format you can store in a file is called pickling. Unpickling is the opposite process, which restores the original Python object from the file.

You can benefit from the pickle module to save your Python objects for later use, send them across a network, or store them in a database. It’s important to keep in mind that Pickle format is Python-specific, so it might not work with other programming languages . Let’s dive into the pickle file usage in Python.

Some common methods you’ll encounter when working with pickle files are pickle.dump(), pickle.dumps(), pickle.load(), and pickle.loads(). The dump() and load() methods work with binary files, while dumps() and loads() work with strings. Here are some basic steps on how to use them:

• Step 1: Import the pickle module using import pickle.
• Step 2: Create the object you want to pickle and store.
• Step 3: Open a file (use binary mode: "wb" for write, "rb" for read) using the with open("file_path", "wb") as f: syntax.
• Step 4: Use pickle.dump() to save the object to the file: pickle.dump(your_object, f).
• Step 5: Use pickle.load() to load the saved object from the binary file into your Python environment.
• Step 6: Profit! You’ve successfully saved and retrieved your Python object using pickle.

Remember that you may need to adjust the protocol, encoding, and other parameters depending on your Python version and requirements, as stated by Stack Overflow. However, in most cases, the pickle module’s default settings should do the trick.

Python Pickle Multiple Objects

You may want to persist multiple objects simultaneously to simplify your workflow. Pickling multiple objects is straightforward and makes it easier for you to manage and structure your data . Let’s explore how you can achieve this using the pickle module.

To pickle and store multiple objects, start by creating a list or another iterable containing all your desired objects. Then, use the pickle.dump() function to serialize and save the entire list to a file.

Here’s an example:

import pickle

obj1 = {"key1": "value1"}
obj2 = [1, 2, 3]
obj3 = "My String"

objects = [obj1, obj2, obj3]

with open("pickled_objects.pkl", "wb") as f:
    pickle.dump(objects, f)

Now that your objects are saved, you’ll want to load them back into your Python script. Use the pickle.load() function to deserialize the saved file and obtain your objects. Remember, your objects were pickled as a list, so you’ll be working with that list structure:

with open("pickled_objects.pkl", "rb") as f:
    loaded_objects = pickle.load(f)

loaded_obj1, loaded_obj2, loaded_obj3 = loaded_objects

By doing this, you will have successfully pickled and loaded multiple objects using Python’s pickle module . Make sure to keep your data organized and use appropriate data structures for easy manipulation.

Python Pickle vs JSON

As you delve into the world of persisting objects in Python, you’ll encounter two popular serialization methods: Pickle and JSON. Understanding their differences will help you choose the right one for your needs. Let’s compare Python Pickle and JSON in terms of serialization format, speed, and security.

Serialization Format: Pickle allows you to serialize objects in a binary format, while JSON uses a text format. Binary means that pickle is not human-readable, unlike JSON, which is simple and easy to parse for humans. If you need to share serialized data between machines, you may want to choose JSON as it’s interoperable and widely used outside the Python ecosystem. Pickle, on the other hand, is Python-specific.

Speed: When it comes to speed and compatibility, Pickle has the upper hand. It can serialize any arbitrary Python object like lists, tuples, and dictionaries, whereas JSON is limited to certain types of data. Additionally, Pickle is generally faster at serialization and deserialization, due to its binary nature.

Security: In terms of security, JSON holds the edge. Pickle can pose security risks if you deserialize data from untrusted sources. JSON is more secure and does not have as many inherent security issues.

Here’s a quick summary of the differences between Python Pickle and JSON:

• Human-readability: JSON ⟷ Pickle
• Interoperability: JSON ⟷ Pickle
• Compatibility: Pickle ⟷ JSON
• Speed: Pickle ⟷ JSON
• Security: JSON ⟷ Pickle

Ultimately, your choice will depend on your specific needs. Consider if human-readability, portability, speed, and security are paramount to your project.

Python Pickle File Extension

When working with the Python pickle module, you’ll need to save the serialized objects to a file. To maintain consistency and make your pickled files easily recognizable, using a specific file extension is crucial. The most common file extensions for pickle files are .pkl and .pickle. Preferred file extensions for pickle vary slightly between Python 2 and Python 3.

For Python 2, it’s recommended to use the .pkl extension when serializing your objects. An example of writing to a pickle file in Python 2 would look like:

output = open('data.pkl', 'wb')
pickle.dump(obj, output)
output.close()

In Python 3, the preferred file extension is .pickle. Here’s an example of how to save a serialized object using the pickle module in Python 3:

with open('data.pickle', 'wb') as f:
    pickle.dump(obj, f)

Ultimately, the choice between .pkl and .pickle as your file extension depends on your Python version and personal preference. Just make sure to stay consistent in your naming conventions to avoid confusion!

Remember that using a widely recognized file extension for your pickled objects helps you and assists others when collaborating on projects.

Python Pickle Save

Saving objects in Python using the pickle module is simple and efficient. Let’s explore the process of saving Python objects for future use.

First, you need to import the pickle module into your Python script:

import pickle

Now, let’s say you have an object you want to save:

my_object = {
    "name": "John Doe",
    "age": 30,
    "hobbies": ["reading", "gaming", "hiking"]
}

To save this object to a file, you need to use the pickle.dump() method along with a file object opened in binary write mode:

with open('my_object.pkl', 'wb') as file:
    pickle.dump(my_object, file)

Congratulations! You’ve just saved your Python object using the pickle module. Remember these key points:

• Always use binary mode when opening files for pickling
• Use pickle.dump() to save an object, passing both the object and a writable binary file
• The file extension can be .pkl or anything, but .pkl is commonly used for pickled files

Keep in mind that the pickle module is perfect for data persistence in Python, but you should be cautious when loading data from untrusted sources, as it may contain security risks.

Python Pickle Dump

Python’s pickle module allows you to easily save and load serialized objects in your code. In this section, we’ll discuss how to use the pickle.dump() method to persist your Python objects to a file.

First, make sure to import the pickle module with the following line of code:

import pickle

Next, you’ll create the object you want to save. It can be a simple data structure such as a list, dictionary, or even a complex custom class instance. For example:

data = {'a': 1, 'b': 2, 'c': 3}

Now, you’re ready to save the data structure using pickle.dump(). Open a file in binary write mode ('wb') and pass it along with the object to be dumped:

with open('data.pkl', 'wb') as file:
    pickle.dump(data, file)

That’s it! You’ve just saved your object to a file named data.pkl.

Keep in mind the following tips while using dump():

• Always open the file in binary mode (with 'wb' or 'rb').
• While not necessary, using the with statement is good practice as it automatically closes the file after the block of code is executed.
• Be aware that not all objects can be pickled. For example, objects with lambdas or open file objects cannot be processed by the pickle module.

Now that you know how to use the Python pickle dump method, you’re ready to persist your Python objects and retrieve them whenever needed. Happy coding!

Recommended: Introduction to Python Classes

The previous article was a theoretical explanation of how I envision this project. It is now time to start laying the groundwork!

I learn as I go…

As some of you might have guessed already, I’m a completely self-taught coder. Because of that, I’m sure many professionals might not agree with how I code.

I accept that and will welcome any constructive criticism. I have been learning non-stop since I started this project. I assume this will not slow down anytime soon. YouTube is my main source of knowledge as I learn best when seeing something done. I am, in other words, a visual learner. I found an article that explains it well here.

Articles on various sites are the other half of how I learn new concepts in coding. That is how I found Finxter :-).

So to sum it up, my code is far from perfect, and I will never claim it is. This is my take on trying to solve this puzzle. I actually look forward to alternative approaches!

You can open issues on my GitHub if you want to address something.

Now that we all know where we stand let us dive right in! How to build a server for our peer-to-peer social network app?

Flask vs FastAPI

In the previous article, I mentioned that I want to use FastAPI to build the relay server, as opposed to Flask. As I have done before and will do again, I asked ChatGPT about the differences between Flask and FastAPI.

 Flask vs FastAPI

Flask is based on the Werkzeug WSGI (Web Server Gateway Interface) toolkit, which is synchronous by default. However, Flask can still be used to build asynchronous applications. You will need to use a third-party library like gevent or asyncio. With these libraries, Flask can use coroutines and event loops to handle I/O operations asynchronously.

FastAPI, on the other hand, is designed to be fully asynchronous from the ground up. It uses the async/await syntax of Python to write asynchronous code. It is based on the ASGI (Asynchronous Server Gateway Interface) specification. FastAPI uses the Starlette framework as its foundation. the framework provides a high-performance event loop and asynchronous request handlers.

Both the speed and the asynchrony determined my choice for FastAPI.

Those of you familiar with Flask will know about its built-in development server. As FastAPI doesn’t have this, we’ll need to install a separate server.

Uvicorn Server

This is where I encountered my first small hiccup. I code on Windows (I know, sue me ), and I wanted to use Uvicorn. As this only runs on Linux, I needed to get it to function in WSL.

I’ll not go into all the details here, but I could write something about it if anyone has an interest in it. Let me know!

After getting Uvicorn to function as it should, we can continue. It is important to remember that the Python interpreter on WSL does not share anything with its Windows counterpart. This means that you either need two separate virtual environments or that you install pip packages for each OS.

Creating Basic FastAPI App

Once all this annoying prep work is done, creating a basic FastAPI app is very easy. We first import FastAPI as below:

from fastapi import FastAPI

All you need to do afterward is define the basic app and create an endpoint.

#---APP INIT---#
app = FastAPI()
 
@app.get("/")
async def root():
    return {"message": "Hello World"}

To get this to run, you need to navigate to the working directory of your FastAPI project via WSL. Afterward, you call the Uvicorn server. The command below assumes you called your Python file main.py!

uvicorn main:app --reload

I usually run the Uvicorn server in a separate terminal instance of WSL.

That way, I can leave it on and test any changes I make immediately. Later, when I’ll be working on the client also, I can split the terminal. You can then make API calls through the client terminal window. FastAPI’s response in the server WSL window is then visible immediately.

Receiving “Hello World” from Server

If you now navigate to 127.0.0.1:8000 you should get a JSON response with the "Hello World" we returned in the endpoint above. We will change this endpoints function later, but for now, it works to prove our API is working.

For the API server, I have the following layout in mind. It might change throughout the development process. I currently foresee two endpoints that do not require the user to be logged in with a JWT token. The first will be to get that token, and the second to register a new user. Everything else will require the user to be authenticated.

I stated earlier that I would change the root’s endpoint function. Its new role is now to allow a user to request a JWT token. The token is only granted after providing a correct combination of username and password. This requires a dedicated set of both helper functions and Pydantic models to work.

I will go into this in another article, as it requires much explaining :-). It was something I am still learning myself.

Endpoint Layout

The current layout of my endpoints at a high level is the following:

#---OPEN ENDPOINTS---#


#Root route to get token
@app.post("/", response_model=Token)


#Route to create a new user
@app.post("/api/v1/users")


#---AUTH ENDPOINTS---#


#Route to get all current users and their attributes(development only)
@app.get("/api/v1/users")


#Route to test if the token is valid, used while authenticating
@app.get("/api/v1/token-test")


#Route to get all thoughts/messages created by a certain user
@app.get("/api/v1/thoughts/{username}")


#Route to return all thoughts/messages containing the query string
@app.get("/api/v1/thoughts/{query_str}")


#Route to create a new message/thought
@app.post("/api/v1/thoughts")


#Route to return all info about the current user(like a user profile)
@app.get("/api/v1/me", response_model=User)

The current setup should allow for the barebones functionality of the application. At least from a server point of view. The routes above and/or their function are liable to change during development.  I do find it helps to have a visual reminder of what I am working toward. That is why I created the high-level outlay. As you might recall, I am a visual learner .

Database Considerations

I will dedicate the last part of this article to the database part. As we need to store both users, user credentials, and messages/tweets somewhere, a database is a must.

If you have read any of my previous articles, you will know I like Deta a lot.

Their NoSQL databases work great for development. They recently evolved into Deta Space. This change makes their ecosystem even more interesting for developers. The fact that they are free is also important for a single developer coding this app on his own time . Make sure to check them out!

The next article will focus on both the database code and the Pydantic models we will need to get our API to function.

Next Article: My Journey to Help Build a P2P Social Network – Database Code Structure

Join GitHub

As always, feel free to ask me questions or pass suggestions! And check out the GitHub repository for participation!

GitHub: https://github.com/shandralor/PeerBrain

This article will show you how to create, update and query a SQLite database using Python.

The SQLite database is a great choice! It is self-contained, easy to use, easy to learn, and works on other devices, such as game consoles, mobile phones, and so on!

To make it more interesting, we have the following running scenario:

KarTech, a used car dealership in Toronto, currently has its inventory of cars saved as a CSV file format. However, they would like this imported into a database. For simplicity, they selected SQLite.

To follow along with this article, download and place the cars.csv file into the current working directory:

Question: How would we write code to create and work with a SQLite Database?

We can accomplish this task by performing the following these steps:

• Preparation
• Declare Variables
• Create SQLite Database and Table
• Retrieve Table Column Names
• Execute Select Query
• Execute Insert Query
• Execute Delete Query
• Close Connections

Preparation

The first step is to import and install the required library to ensure the code in this article will run error-free.

If the pandas library is not installed. Click here for instructions.

Next, create a Python file in the current working directory called cars-sql.py and paste the following code to the top of this file.

import pandas as pd
import sqlite3
from os.path import exists as file_exists

The first line in the above code snippet calls in the pandas library, which is required to access and manipulate the cars.csv file downloaded earlier.

The following line imports Python’s built-in sqlite3 library required to access and manipulate the SQLite database.

The last line also imports Python’s built-in os library that checks for the existence of the Database file.

Save this file.

Declare Variables

This section creates and assigns variables for the Database, its associated Table, and the cars.csv file downloaded earlier.

Paste the following lines to the bottom of the cars-sql.py file.

dbase_name = 'cars.db'
table_name = 'inventory'
csv_file = 'cars.csv'

Save this file.

Create Database and Table

This section does all the heavy lifting! It creates a Database if it does not already exist and attempts to connect to the same!

Paste the following lines to the bottom of the cars-sql.py file.

if file_exists(dbase_name):
    conn = sqlite3.connect(dbase_name)
else:
    conn = sqlite3.connect(dbase_name)
    df = pd.read_csv('cars.csv', sep=';')
    df.to_sql(f'{table_name}', conn)
cur = conn.cursor()

The first line in the above code snippet checks to see if the Database exists in the current working directory (cars.db). If this file exists, a Connection Object is created (conn), allowing access to the Database (shown below).

Connection Object: <sqlite3.Connection object at 0x000001E8653DAB40>

If the Database does not exist, code execution falls inside the else statement where the following occurs:

• A Database is created, and a Connection object to the same saves to conn.
• The cars.csv file is read in (notice the separator character). The results save to the DataFrame, df.
• Then the contents of df are imported into the table specified earlier (inventory).

The last line creates a Cursor Object (cur) from the Connection Object (conn). The Cursor Object allows access to commands, such as SQL, which can be applied against the Database and/or Table.

Cursor Object: <sqlite3.Cursor object at 0x0000024D38B562C0>

Save this file.

If successful, the cars.db SQLite database should now reside in the current working directory.

Retrieve Table Column Names

This section shows how to retrieve a list of the Column Names in the Table.

Paste the following lines to the bottom of the cars-sql.py file.

cols = conn.execute('SELECT * FROM inventory')
conn.commit()

col_names = list(map(lambda x: x[0], cols.description))

The first two (2) lines in the above code snippet execute and commit a SQL command that receives all rows from the inventory Table. A Cursor Object is created from these results and saved to cols. If output to the terminal, the following would display.

Cols Object: <sqlite3.Cursor object at 0x0000022582C9CBC0>

The following lines access and retrieve the cols.description field using a lambda, a map(), and a list. This action retrieves the Field Names from the Table and saves the results to tbl_fields. If output to the terminal, the following would display.

['index', 'Car', 'MPG', 'Cylinders', 'Displacement', 'Horsepower', 'Weight', 'Acceleration', 'Model', 'Origin']

Save this file.

Note: The commit() command saves changes invoked by a SQL query to the database. Click here to learn more.

Execute Select Query

This section shows how to create and execute a SELECT query.

Paste the following lines to the bottom of the cars-sql.py file.

When executing a SELECT query, this task can be performed using the read_sql() function and passing it two (2) arguments, a SQL statement and a Connection Object (conn).

query = pd.read_sql(f'SELECT Car, MPG, Cylinders FROM {table_name} LIMIT 5', conn)

This SQL query selects and retrieves three (3) columns and five (5) rows from the inventory table. The results save to query. If output to the terminal, the following would display.

   Car                                                    MPG      Cylinders
0  Chevrolet Chevelle Malibu                18.0             8
1  Buick Skylark 320                             15.0             8
2  Plymouth Satellite                             18.0             8
3  AMC Rebel SST                                16.0             8
4  Ford Torino                                        17.0             8

Save this file.

Note: The above method is commonly used for SELECT queries as we are not altering the Database or the Table, such as performing an update, insert, delete, drop, or alter command. Click here to learn more about SQL.

Execute Insert Query

This section outlines how to create and execute an INSERT query

Paste the following lines to the bottom of the cars-sql.py file.

cur.execute('''INSERT INTO inventory
             VALUES('Citroen', '28.0', '8', '', '', '', '', '', '', '')''')

print(cur.lastrowid)
conn.commit()

The first line in the above code snippet creates and executes a SQL statement that INSERTS one (1) record into the inventory Table.

The following line retrieves the Row ID and is output to the terminal. If this returns an integer value, then the INSERT was successful.

408

Execute Delete Query

This section outlines how to create and execute a DELETE query

Paste the following lines to the bottom of the cars-sql.py file.

cur.execute('''DELETE FROM inventory WHERE MPG = 28.0''')
conn.commit()

The first line in the above code snippet creates and executes a SQL statement that DELETES all records matching the criteria from the inventory Table.

The following line commits the changes to the Database.

Note: Run a SELECT query to see if the record(s) were successfully deleted.

Close the Connections

The final step is to close any open connections.

Paste the following lines to the bottom of the cars-sql.py file.

cur.close()
conn.close()

The Entire Script

import pandas as pd
import sqlite3
from os.path import exists as file_exists

dbase_name = 'cars.db'
table_name = 'inventory'
csv_file = 'cars.csv'

if file_exists(dbase_name):
    conn = sqlite3.connect(dbase_name)
else:
    conn = sqlite3.connect(dbase_name)
    df = pd.read_csv('cars.csv', sep=';')
    df.to_sql(f'{table_name}', conn)
cur = conn.cursor()


# Get Column Names
cols = conn.execute('SELECT * FROM inventory')
conn.commit()
col_names = list(map(lambda x: x[0], cols.description))

# SELECT
query = pd.read_sql(f'SELECT Car, MPG, Cylinders FROM {table_name} LIMIT 5', conn)
print(query)

# INSERT
cur.execute('''INSERT INTO inventory
             VALUES('Citroen', '28.0', '8', '', '', '', '', '', '', '')''')

print(cur.lastrowid)
conn.commit()

# DELETE
cur.execute('''DELETE FROM inventory WHERE MPG = 28.0''')
conn.commit()

cur.close()
conn.close()

Summary

This article has shown you how to work with a SQLite Database.

Good Luck & Happy Coding!

1. Establish database connection,
2. Run the SQL query to select data,
3. Store the data in the CSV using basic Python file I/O and the csv module, and
4. Close the database connection.

Here’s a specific example with additional annotations and simplified for clarity:

import pyodbc
import csv


# 1. Example establish database connection
connection = pyodbc.connect("your connection string")

# 2. Run SQL query
cursor = connection.cursor()
cursor.execute('select * from XXX;')

# 3. Store the contents in "cursor" in the CSV using file I/O
with open('my_file.csv','w') as f:
    writer = csv.writer(f)
    writer.writerows([x[0] for x in cursor.description])
    writer.writerows(cursor)

# 4. Close the database connection
cursor.close()
conn.close()

Basically, as soon as you have mastered the art of pulling data into the cursor variable using the pyodbc module, you can use all different ways to write that data into the CSV using basic Python file I/O — such as the open() function or context managers — or even Pandas function to write DataFrames to CSV files.

If you’ve landed on this article, you’ll probably struggle with one or both of those issues:

• Understanding basic Python features and functions—or knowing about them in the first place. In that case, join my free email academy for an infinite stream of learning content.
• Understanding the database handling provided by Python’s Access interface. In that case, check out our in-depth tutorial on pyodbc on the Finxter blog.

I hope you learned something out of today’s tutorial, my friend! Here’s a joke to round things out:

Nerd Humor

The following career paths for coders interested in the broad database space are ordered alphabetically. The table shows the annual income of different job descriptions in the database space:

Let’s dive into each of those job descriptions next!

Cassandra Developer

Cassandra is a free, open-source, distributed, NoSQL database management system (DBMS) for large amounts of data that may be distributed across many commodity servers.

A Cassandra developer manages the Cassandra system and integrates applications with the database storage solution and API.

The average annual income of a Cassandra Developer is between $110,000 and $145,500, according to Glassdoor (source).

Do you want to become a Cassandra Developer? Here’s a learning path I’d propose in five steps to get started:

• Step 1: Introduction to Computer Science (~40 hours)
• Step 2: Introduction to Databases (~20 hours)
• Step 2: Introduction to Distributed Systems (~10 hours)
• Step 3: Introduction to Apache Cassandra (~20 hours)

Learn More: Feel free to read my full guide on this career role on this Finxter blog article.

Couchbase Developer

Couchbase is an open-source, distributed NoSQL database infrastructure for interactive applications to serve concurrent users.

A Couchbase developer creates applications using this infrastructure—mostly for business clients.

The average Couchbase salary ranges from approximately $87,504 per year for an Inside Sales Representative to $212,621 per year for a Senior Software Engineer (source).

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

Database Administrator

A database administrator (DBA) is an IT professional responsible for maintaining the reliable, efficient, and secure execution of a database management system (DBMS).

In this way, a database administrator is responsible for providing the data infrastructure of a company or organization. This involves installing, configuring, debugging, optimizing, securing, troubleshooting, and managing database systems.

DBAs can either work as employees or as freelancers remotely or onsite.

The average annual income of a Database Administrator in the United States is between $83,533 and $131,056, with an average of $100,920 and a statistical median of $98,860 per year.

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

Database Developer

A database engineer is responsible for providing the data infrastructure of a company or organization. This involves designing, creating, installing, configuring, debugging, optimizing, securing, and managing databases.

Database engineers can either work as employees or as freelancers remotely or onsite.

A database engineer has the following responsibilities:

1. Creating a new database system.
2. Finding a database system tailored to the needs of an organization.
3. Designing the data models.
4. Accessing the data with scripting languages including SQL-like syntax.
5. Installing an existing database software system onsite.
6. Configuring a database system.
7. Optimizing a database management system for performance, speed, or reliability.
8. Consulting management regarding data management issues.
9. Keeping databases secure and providing proper access control to users.
10. Monitoring and managing an existing database system to keep it running smoothly.
11. Debugging potential bugs, errors, and security issues detected at runtime.
12. Testing and deploying a database system on a public cloud infrastructure such as AWS.
13. Handling distribution issues in the case of a distributed database management system.
14. Ensuring budget adherence when running on a public cloud and estimating costs for private database solutions.
15. Communicating and negotiating with salespeople (e.g., from Oracle).

The average annual income of a Database Engineer in the United States is between $72,536 and $135,000, with an average of $103,652 and a statistical median of $106,589 per year.

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

DynamoDB Developer

Amazon’s DynamoDB is a NoSQL, serverless, key-value database for high-performance applications. It is fully managed so a DynamoDB developer usually helps clients set up and connect to the database—and provides minimal maintenance support.

As a DynamoDB Developer, you can expect to earn between $70,000 and $160,000 per year according to Google (source).

Do you want to become a DynamoDB Developer?

Here’s a learning path I’d propose in five steps to get started:

• Step 1: Introduction to Computer Science (~40 hours)
• Step 2: Introduction to Distributed Systems (~20 hours)
• Step 3: Introduction to Databases (~10 hours)
• Step 4: Introduction to DynamoDB (~10 hours)
• Step 5: Introduction to Web Services (~10 hours)

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

IBM DB2 Developer

Note: If I may be so blunt—I don’t recommend this career path anymore due to the “dying” interest in IBM DB2 technology.

IBM DB2 is a data processing and storage solution that helps developers quickly build and deploy applications such as natural language querying and AI. An IBM DB2 developer creates apps for and integrates them into this system.

The average annual income of an IBM DB2 Developer is $132,000 according to Glassdoor (source).

Learn More: Feel free to read my full guide on this career role on this Finxter blog article.

MariaDB Developer

MariaDB is a database framework focusing on relational databases that are compatible with Oracle. MariaDB developers provide value to clients by integrating their applications with new or existing MariaDB databases. (Source)

The average annual income of a MariaDB Developer is $85,000 according to Payscale (source).

Learn More: Feel free to read my full guide on this career role on this Finxter blog article.

Microsoft SQL Server Developer

Microsoft SQL Server is a relational database management system (RDBMS) to store and retrieve data on the same computer or across a network. A Microsoft SQL Server Developer creates and connects applications that access the Microsoft SQL Server API. (Source)

The average annual income of a Microsoft SQL Server Developer is between $74,000 and $137,500 according to Ziprecruiter (source):

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

MongoDB Developer

The average annual income of a MongoDB Developer is $92,414 with a most likely range between $59,000 and $146,000 and a top earner income of $222,000 according to Glassdoor (source).

MongoDB is a source-available cross-platform document-oriented database program. As a NoSQL database program, MongoDB uses JSON-like documents with optional schemas.

A MongoDB developer implements, maintains, and designs MongoDB database applications. (Source)

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

MS Access Developer

The average annual income of an MS Access Developer is between $45,000 (25th percentile) and $75,000 (75th percentile) according to Ziprecruiter with top developers making $122,000 and more (source).

A Microsoft Access Developer developer creates, edits, analyzes, debugs, and supervises the development of software written in the MS Access programming language.

MS Access is a database system. Although it isn’t used that much anymore, it is still a great tool for small projects. And there is a huge number of legacy systems that depend upon MS Access. (Source)

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

MySQL Developer

A MySQL Developer is responsible for database administration, such as performance optimization, troubleshooting, problem-solving, and customer support.

The average annual income of a MySQL Developer is between $87,828 and $149,975, according to Glassdoor (source).

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

Oracle Developer

An Oracle developer creates or maintains the database components of an Oracle application. They often develop new applications or convert existing applications to run in an Oracle Database environment. (Source)

The average annual income of an Oracle Developer in the United States is between $86,074 and $107,675, according to Salary.com, with an average of $97,941. (source)

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

PL/SQL Developer

A PL-/SQL Developer developer creates, edits, analyzes, debugs, and supervises the development of software written in the PL/SQL programming language.

PL/SQL is a procedural language built on top of SQL for programming Oracle databases. (Source)

The average annual income of a PL/SQL Developer is between $72,000 and $103,000 according to Glassdoor (source).

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

PostgreSQL Developer

PostgreSQL calls itself “The World’s Most Advanced Open Source Relational Database”.

A PostgreSQL database developer works on the PostgreSQL project whereas a PostgreSQL developer works with PostgreSQL on a third-party application. Most will take the second route, for which you need to know how to access the API. (Source)

Do you want to become a PostgreSQL developer? Here’s a learning path I’d propose in five steps to get started:

• Step 1: Introduction to Computer Science (~40 hours)
• Step 2: Introduction to Databases (~10 hours)
• Step 3: Introduction to SQL (~10 hours)
• Step 4: Introduction to PostgreSQL (~10 hours)

According to ZipRecruiter, the annual income of PostgreSQL developers is between $104,000 for the bottom 25th percentile, $133,000 for the 75th Percentile, and $171,500 for the top earners.

The monthly pay ranges from $8,666 to $14,291 with an average monthly income of $10,052.

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

RavenDB Developer

The average annual income of a RavenDB Developer is hard to estimate given the limited data availability. However, RavenDB developers are NoSQL developers who earn between $101,000 (25th percentile) and $130,500 (75th percentile) with an average of $119,105 per year in the US according to Ziprecruiter (source).

A RavenDB Developer developer creates, edits, analyzes, debugs, and supervises the development of software written in the RavenDB programming language. RavenDB is a NoSQL document-oriented database written especially for the .NET framework. (Source)

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

Redis Developer

Redis (Remote Dictionary Server) is a fast, open source, in-memory, key-value data store.

As a Redis developer, you will apply and develop Redis solutions for databases, caches, message brokers, and queues.

Other use cases are chat apps, gaming leaderboards, session stores, streaming apps, and handling geospatial data sources — to name just a few. (Source)

The average annual income of a Redis developer is $113,000 in the US according to PayScale (source).

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

SQL Developer

An SQL Developer creates, among other things, SQL code for database access. SQL is short for Structured Query Language that is designed to manage data stored in relational databases. (Source)

The average annual income of a SQL Developer in the US is $76,110 according to PayScale and $87,489 according to Indeed.com.

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

SQLite Developer

SQLite is the most used database engine in the world. SQLite implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine.

SQLite developers help clients set up, maintain, and manage an SQLite database for their applications. (Source)

The average annual income of a SQLite Developer is between $49,000 and $107,000 with an average of $64,977 per year according to PayScale (source).

Learn More: Feel free to read my full guide on this career role in this Finxter blog article.

Summary

The most attractive, common, or interesting career paths as a database enthusiast are (top 18):

1. Cassandra Developer
2. Couchbase Developer
3. Database Administrator
4. Database Developer
5. DynamoDB Developer
6. IBM DB2 Developer
7. MariaDB Developer
8. Microsoft SQL Server Developer
9. MongoDB Developer
10. MS Access Developer
11. MySQL Developer
12. Oracle Developer
13. PL-SQL Developer
14. PostgreSQL Developer
15. RavenDB Developer
16. Redis Developer
17. SQL Developer
18. SQLite Developer

Feel free to check out our full career guide on the Finxter blog and the free Finxter email academy with never-ending free content!

What is a Database Administrator?

A database administrator (DBA) is an IT professional responsible for maintaining the reliable, efficient, and secure execution of a database management system (DBMS).

In this way, a database administrator is responsible for providing the data infrastructure of a company or organization. This involves installing, configuring, debugging, optimizing, securing, troubleshooting, and managing database systems.

DBAs can either work as employees or as freelancers remotely or onsite.

What Does a Database Administrator Do?

As already indicated, a database admin is responsible for providing and maintaining the data infrastructure of a company or organization.

In particular, a database administrator has many responsibilities, such as the following 15 most crucial activities handled by DBAs today:

1. Migrating data from one database to another
2. Designing the data models.
3. Accessing the data with scripting languages including SQL-like syntax.
4. Finding a database system tailored to the needs of an organization.
5. Installing an existing database software system onsite.
6. Configuring a database system.
7. Optimizing a database management system for performance, speed, or reliability.
8. Consulting management regarding data management issues.
9. Keeping databases secure and providing proper access control to users.
10. Monitoring and managing an existing database system to keep it running smoothly.
11. Debugging potential bugs, errors, and security issues detected at runtime.
12. Testing and deploying a database system on a public cloud infrastructure such as AWS.
13. Handling distribution issues in the case of a distributed database management system.
14. Ensuring budget adherence when running on a public cloud and estimating costs for private database solutions.
15. Communicating and negotiating with salespeople (e.g., from Oracle).

These are only some of the most common activities frequently handled by DBAs.

Database Administrator vs Data Engineer

A database administrator is responsible for providing and maintaining the data infrastructure of a company or organization. This involves installing, configuring, debugging, optimizing, securing, troubleshooting, and managing databases. DBAs can either work as employees or as freelancers remotely or onsite.

A data engineer prepares data to be used in data analytics and operations, essentially providing automated or semi-automated ways for data collection and creating pipelines that connect various data sources to database management systems such as the ones managed by a DBA.

A data engineer focuses on filling data into a database system whereas a database admin is focused on maintaining the database system in the first place. There are intersection points between data engineers and DBAs at the interface between data sources and data management.

Database Administrator vs Database Engineer

A database engineer is responsible for providing the data infrastructure of a company or organization. This involves designing, creating, installing, configuring, debugging, optimizing, securing, and managing databases. Database engineers can either work as employees or as freelancers remotely or onsite.

Database administrators perform a similar role to database engineers in that they are responsible for setting up, installing, configuring, securing, and managing a database management system.

The focus of DBAs is more on the technical maintenance of existing systems rather than the theoretical development of new solutions.

But the lines between those two job descriptions are blurry and often overlap significantly.

Annual Income of Database Administrator (US)

Question: How much does a Database Administrator (DBA) in the US make per year?

The average annual income of a Database Administrator in the United States is between $83,533 and $131,056, with an average of $100,920 and a statistical median of $98,860 per year.

This data is based on our meta-study of 13 salary aggregators sources such as Glassdoor, ZipRecruiter, and PayScale.

Note: This is the most comprehensive salary meta-study of database administrator income in the world, to the best of my knowledge!

Further note that DBAs earn less compared to database engineers, although many of the tasks of a DBA are similar to those of a database engineer:

To learn more about the income potential and opportunity of data engineers, have a look at our full guide on the Finxter blog:

• Data Engineer — Income and Opportunity

Let’s have a look at the hourly rate of DBAs next!

Hourly Rate

Database Administrators are well-paid on freelancing platforms such as Upwork or Fiverr.

• Related Article: What’s the best freelancing platform?

If you decide to go the route as a freelance database administrator, you can expect to make between $20 and $120 per hour on Upwork (source). Assuming an annual workload of 2000 hours, you can expect to make between $40,000 and $240,000 per year.

Note that it may be a smart trick to reposition yourself as a database engineer to instantly make 10-20% more! Learn more here on the Finxter blog.

Note: Do you want to create your own thriving coding business online? Feel free to check out our freelance developer course — the world’s #1 best-selling freelance developer course that specifically shows you how to succeed on Upwork and Fiverr!

Industry Demand

But is there enough demand? Let’s have a look at Google trends to find out how interest evolves over time (source):

The interest in database administration has dropped by almost 75% over the last two decades — with more and more solutions centralizing in the cloud.

If you compare the interest of “database engineering” with “database administration”, you can see that “database engineering” actually wins in relative importance (source) because it doesn’t shrink as quickly:

Learning Path, Skills, and Education Requirements

Do you want to become a Database Admin? Here’s a step-by-step learning path I’d propose to get started:

• Step 1: Introduction to Computer Science (~40 hours)
• Step 2: Introduction to Distributed Systems (~20 hours)
• Step 3: Introduction to Databases (Free Video Courses) (~10 hours)
• Step 4: Introduction to MongoDB (~10 hours)
• Step 5: Introduction to SQL (~10 hours)
• Step 6: Introduction to MariaDB (~10 hours)

Here you can already start with the first step — do it now!

You can find many additional computer science courses on the Finxter Computer Science Academy (flatrate model).

But don’t wait too long to acquire practical experience!

Even if you have little skills, it’s best to get started as a freelance developer and learn as you work on real projects for clients — earning income as you learn and gaining motivation through real-world feedback.

Tip: An excellent start to turbo-charge your freelancing career (earning more in less time) is our Finxter Freelancer Course. The goal of the course is to pay for itself!

Related Video

You can find more job descriptions for coders, programmers, and computer scientists in our detailed overview guide:

• Get Your Job in IT [Overview]

Related Income of Professional Developers

The following statistic shows the self-reported income from 9,649 US-based professional developers (source).

The average annual income of professional developers in the US is between $70,000 and $177,500 for various programming languages.

Question: What is your current total compensation (salary, bonuses, and perks, before taxes and deductions)? Please enter a whole number in the box below, without any punctuation. If you are paid hourly, please estimate an equivalent weekly, monthly, or yearly salary. (source)

The following statistic compares the self-reported income from 46,693 professional programmers as conducted by StackOverflow.

The average annual income of professional developers worldwide (US and non-US) is between $33,000 and $95,000 for various programming languages.

Here’s a screenshot of a more detailed overview of each programming language considered in the report:

Here’s what different database professionals earn:

Here’s an overview of different cloud solutions experts:

Here’s what professionals in web frameworks earn:

There are many other interesting frameworks—that pay well!

Look at those tools:

Okay, but what do you need to do to get there? What are the skill requirements and qualifications to make you become a professional developer in the area you desire?

Let’s find out next!

General Qualifications of Professionals

StackOverflow performs an annual survey asking professionals, coders, developers, researchers, and engineers various questions about their background and job satisfaction on their website.

Interestingly, when aggregating the data of the developers’ educational background, a good three quarters have an academic background.

Here’s the question asked by StackOverflow (source):

Which of the following best describes the highest level of formal education that you’ve completed?

However, if you don’t have a formal degree, don’t fear! Many of the respondents with degrees don’t have a degree in their field—so it may not be of much value for their coding careers anyways.

Also, about one out of four don’t have a formal degree and still succeeds in their field! You certainly don’t need a degree if you’re committed to your own success!

Freelancing vs Employment Status

The percentage of freelance developers increases steadily. The fraction of freelance developers has already reached 11.21%!

This indicates that more and more work will be done in a more flexible work environment—and fewer and fewer companies and clients want to hire inflexible talent.

Here are the stats from the StackOverflow developer survey (source):

Do you want to become a professional freelance developer and earn some money on the side or as your primary source of income?

Resource: Check out our freelance developer course—it’s the best freelance developer course in the world with the highest student success rate in the industry!

Other Programming Languages Used by Professional Developers

The StackOverflow developer survey collected 58000 responses about the following question (source):

Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?

These are the languages you want to focus on when starting out as a coder:

And don’t worry—if you feel stuck or struggle with a nasty bug. We all go through it. Here’s what SO survey respondents and professional developers do when they’re stuck:

What do you do when you get stuck on a problem? Select all that apply. (source)

Related Tutorials

To get started with some of the fundamentals and industry concepts, feel free to check out these articles:

• Freelance Developer – How to Code From Home and Earn Six Figures [Industry Report]
• How to Become a Python Freelancer—and Earn $1,000 on the Side? [A Step-by-Step Tutorial]
• How Adam Earns $5000 per Month as a Python Freelancer on Upwork [Month 4]
• Database Wikipedia
• Learn Database Google

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

References

[1] The figure was generated using the following code snippet:

import matplotlib.pyplot as plt
import numpy as np
import math

data = [91541,
        107844,
        120961,
        135000,
        106037,
        88419,
        107141,
        110987,
        96058,
        72536]

labels = ['Glassdoor.com',
          'ZipRecruiter.com',
          'BuiltIn.com',
          'Talent.com',
          'Indeed.com',
          'PayScale.com',
          'SalaryExpert.com',
          'Comparably.com',
          'Zippia.com',
          'Salary.com']

median = np.median(data)
average = np.average(data)
print(median, average)
n = len(data)

plt.plot(range(n), [median] * n, color='black', label='Median: $' + str(int(median)))
plt.plot(range(n), [average] * n, '--', color='red', label='Average: $' + str(int(average)))
plt.bar(range(len(data)), data)
plt.xticks(range(len(data)), labels, rotation='vertical', position = (0,0.45), color='white', weight='bold')
plt.ylabel('Average Income ($)')
plt.title('Database Engineer Annual Income - by Finxter')
plt.legend()
plt.show()

What is a Database Engineer?

A database engineer is responsible for providing the data infrastructure of a company or organization. This involves designing, creating, installing, configuring, debugging, optimizing, securing, and managing databases. Database engineers can either work as employees or as freelancers remotely or onsite.

What Does a Database Engineer Do?

As already indicated, a database engineer is responsible for providing the data infrastructure of a company or organization.

In particular, a database engineer has many responsibilities, such as the following 15 most popular activities performed by a database engineer today:

1. Creating a new database system.
2. Finding a database system tailored to the needs of an organization.
3. Designing the data models.
4. Accessing the data with scripting languages including SQL-like syntax.
5. Installing an existing database software system onsite.
6. Configuring a database system.
7. Optimizing a database management system for performance, speed, or reliability.
8. Consulting management regarding data management issues.
9. Keeping databases secure and providing proper access control to users.
10. Monitoring and managing an existing database system to keep it running smoothly.
11. Debugging potential bugs, errors, and security issues detected at runtime.
12. Testing and deploying a database system on a public cloud infrastructure such as AWS.
13. Handling distribution issues in the case of a distributed database management system.
14. Ensuring budget adherence when running on a public cloud and estimating costs for private database solutions.
15. Communicating and negotiating with salespeople (e.g., from Oracle).

These are only some of the most common activities frequently handled by database engineers.

Database Engineer vs Data Engineer

A database engineer is responsible for providing the data infrastructure of a company or organization. This involves designing, creating, installing, configuring, debugging, optimizing, securing, and managing databases. Database engineers can either work as employees or as freelancers remotely or onsite.

A data engineer prepares data to be used in data analytics and operations, essentially providing automated or semi-automated ways for data collection and creating pipelines that connect various data sources to database management systems such as the ones managed by a database engineer.

A data engineer focuses on filling data into a database system whereas a database engineer is focused on providing the database system in the first place. There are intersection points between data engineers and database engineers at the interface between data sources and data management.

Database Engineer vs Database Administrator

Database administrators perform a similar role to database engineers in that they are responsible for setting up, installing, configuring, securing, and managing a database management system.

The focus is more on the technical maintenance of existing systems than the theoretical development of new solutions.

But the lines between those two job descriptions are blurry and often overlap significantly.

Annual Income of Database Engineer (US)

How much does a Database Engineer make per year?

Question: How much does a Database Engineer in the US make per year?

The average annual income of a Database Engineer in the United States is between $72,536 and $135,000, with an average of $103,652 and a statistical median of $106,589 per year.

This data is based on our meta-study of ten (10) salary aggregators sources such as Glassdoor, ZipRecruiter, and PayScale.

Note: This is the most comprehensive salary meta-study of database engineer income in the world, to the best of my knowledge!

Let’s have a look at the hourly rate of Database Engineers next!

Hourly Rate

Database Engineers are well-paid on freelancing platforms such as Upwork or Fiverr.

• Related Article: What’s the best freelancing platform?

If you decide to go the route as a freelance Database Developer, you can expect to make between $30 and $130 per hour on Upwork (source). Assuming an annual workload of 2000 hours, you can expect to make between $60,000 and $260,000 per year.

Note: Do you want to create your own thriving coding business online? Feel free to check out our freelance developer course — the world’s #1 best-selling freelance developer course that specifically shows you how to succeed on Upwork and Fiverr!

Industry Demand

But is there enough demand? Let’s have a look at Google trends to find out how interest evolves over time (source):

The interest in database engineering has remained relatively stable over the last two decades.

If you compare the interest with “database administration”, you can see that “database engineering” actually wins in relative importance (source):

Learning Path, Skills, and Education Requirements

Do you want to become a Database Endineer? Here’s a step-by-step learning path I’d propose to get started with Database :

• Step 1: Introduction to Computer Science (~40 hours)
• Step 2: Introduction to Distributed Systems (~20 hours)
• Step 3: Introduction to Databases (Free Video Courses) (~10 hours)
• Step 4: Introduction to MongoDB (~10 hours)
• Step 5: Introduction to SQL (~10 hours)
• Step 6: Introduction to MariaDB (~10 hours)

Here you can already start with the first step — do it now!

You can find many additional computer science courses on the Finxter Computer Science Academy (flatrate model).

But don’t wait too long to acquire practical experience!

Even if you have little skills, it’s best to get started as a freelance developer and learn as you work on real projects for clients — earning income as you learn and gaining motivation through real-world feedback.

Tip: An excellent start to turbo-charge your freelancing career (earning more in less time) is our Finxter Freelancer Course. The goal of the course is to pay for itself!

Related Video

You can find more job descriptions for coders, programmers, and computer scientists in our detailed overview guide:

• Get Your Job in IT [Overview]

Related Income of Professional Developers

The following statistic shows the self-reported income from 9,649 US-based professional developers (source).

The average annual income of professional developers in the US is between $70,000 and $177,500 for various programming languages.

Question: What is your current total compensation (salary, bonuses, and perks, before taxes and deductions)? Please enter a whole number in the box below, without any punctuation. If you are paid hourly, please estimate an equivalent weekly, monthly, or yearly salary. (source)

The following statistic compares the self-reported income from 46,693 professional programmers as conducted by StackOverflow.

The average annual income of professional developers worldwide (US and non-US) is between $33,000 and $95,000 for various programming languages.

Here’s a screenshot of a more detailed overview of each programming language considered in the report:

Here’s what different database professionals earn:

Here’s an overview of different cloud solutions experts:

Here’s what professionals in web frameworks earn:

There are many other interesting frameworks—that pay well!

Look at those tools:

Okay, but what do you need to do to get there? What are the skill requirements and qualifications to make you become a professional developer in the area you desire?

Let’s find out next!

General Qualifications of Professionals

StackOverflow performs an annual survey asking professionals, coders, developers, researchers, and engineers various questions about their background and job satisfaction on their website.

Interestingly, when aggregating the data of the developers’ educational background, a good three quarters have an academic background.

Here’s the question asked by StackOverflow (source):

Which of the following best describes the highest level of formal education that you’ve completed?

However, if you don’t have a formal degree, don’t fear! Many of the respondents with degrees don’t have a degree in their field—so it may not be of much value for their coding careers anyways.

Also, about one out of four don’t have a formal degree and still succeeds in their field! You certainly don’t need a degree if you’re committed to your own success!

Freelancing vs Employment Status

The percentage of freelance developers increases steadily. The fraction of freelance developers has already reached 11.21%!

This indicates that more and more work will be done in a more flexible work environment—and fewer and fewer companies and clients want to hire inflexible talent.

Here are the stats from the StackOverflow developer survey (source):

Do you want to become a professional freelance developer and earn some money on the side or as your primary source of income?

Resource: Check out our freelance developer course—it’s the best freelance developer course in the world with the highest student success rate in the industry!

Other Programming Languages Used by Professional Developers

The StackOverflow developer survey collected 58000 responses about the following question (source):

Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?

These are the languages you want to focus on when starting out as a coder:

And don’t worry—if you feel stuck or struggle with a nasty bug. We all go through it. Here’s what SO survey respondents and professional developers do when they’re stuck:

What do you do when you get stuck on a problem? Select all that apply. (source)

Related Tutorials

To get started with some of the fundamentals and industry concepts, feel free to check out these articles:

• Freelance Developer – How to Code From Home and Earn Six Figures [Industry Report]
• How to Become a Python Freelancer—and Earn $1,000 on the Side? [A Step-by-Step Tutorial]
• How Adam Earns $5000 per Month as a Python Freelancer on Upwork [Month 4]
• Database Wikipedia
• Learn Database Google

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

References

[1] The figure was generated using the following code snippet:

import matplotlib.pyplot as plt
import numpy as np
import math

data = [91541,
        107844,
        120961,
        135000,
        106037,
        88419,
        107141,
        110987,
        96058,
        72536]

labels = ['Glassdoor.com',
          'ZipRecruiter.com',
          'BuiltIn.com',
          'Talent.com',
          'Indeed.com',
          'PayScale.com',
          'SalaryExpert.com',
          'Comparably.com',
          'Zippia.com',
          'Salary.com']

median = np.median(data)
average = np.average(data)
print(median, average)
n = len(data)

plt.plot(range(n), [median] * n, color='black', label='Median: $' + str(int(median)))
plt.plot(range(n), [average] * n, '--', color='red', label='Average: $' + str(int(average)))
plt.bar(range(len(data)), data)
plt.xticks(range(len(data)), labels, rotation='vertical', position = (0,0.45), color='white', weight='bold')
plt.ylabel('Average Income ($)')
plt.title('Database Engineer Annual Income - by Finxter')
plt.legend()
plt.show()

What Is a Full-Stack Web Developer?

A full-stack web developer works both with back-end and front-end web technologies.

• The back-end consists of the webserver infrastructure, databases, and code function integration to facilitate a smooth and secure serving of user requests.
• The front-end focuses on the graphical user interface (GUI) of the website using HTML, CSS, and JavaScript with the goal of setting up the whole technology stack to enable users to view and interact with the website.

Full-stack developers have skills in all those fields so they often take crucial roles in overseeing the technical implementation of large web projects.

A web developer is a programmer who specializes in the development of websites or applications viewed on web browsers, mobile devices, and large desktop screens that are transported over private or public networks such as the Internet.

A back-end web developer focuses on the implementation and deployment of the “back-end” of a web application. The back-end consists of the webserver infrastructure, databases, and code function integration to facilitate a smooth and secure serving of user requests. Back-end developers often operate behind the scenes (from a user perspective) but work closely in teams with database administrators, distributed systems experts and front-end web developers.

A front-end web developer focuses on the graphical user interface (GUI) of the website using HTML, CSS, and JavaScript with the goal of setting up the whole technology stack to enable users to view and interact with the website.

This video nicely explains some of the most important technologies and skills you need as a back-end web developer:

And this video nicely explains some of the most important technologies and skills you need as a front-end web developer:

Who Do Full-Stack Web Developers Work For?

Full-stack web developers either work independently as freelancers or as employees for companies, government organizations, crypto projects, or non-profits.

Lately, many full-stack web developers have started to work for decentralized autonomous organizations (DAOs) in the crypto ecosystem due to their expertise in native web technologies, distributed systems, databases, and web philosophies.

In those web3 projects, both front-end and back-end skills are needed and full-stack developers can shine!

In fact, the whole Blockchain technology stack is built around “back-end” web technologies—the Blockchain can be seen as a distributed, decentralized, and secure database that is replicated among thousands of participants. It’s the world of back-end web developers!

Almost all decentralized projects (including Bitcoin and Ethereum) only focus on decentralized back-end development whereas the front-ends run on a centralized infrastructure!

Almost all decentralized projects (including Ethereum) only focus on decentralized back-end development whereas the front-ends run on a centralized infrastructure!

That’s why decentralized web3 apps still require front-end skills such as HTML, CSS, and JavaScript.

Annual Income

How much does a Full-Stack Web Developer make per year?

The average annual income of a Full-Stack Web Developer in the United States is between $79,584 and $108,984 with an average income of $98,454 and a median income of $99,274 per year according to our meta-study of 7 aggregated data sources such as Glassdoor and Indeed.

So, a full-stack developer tends to make 2-10% more income as either a back-end or a front-end developer due to the increased breadth of required technical skills.

• Back-end web developers make on average $96,545 (median: $97,522).
• Front-end web developers make on average $89,683 (median: $90,499) per year.
• Full-Stack web developers make on average $98,454 (median: $99,274) per year.

If you need the raw data for the income of a back-end web developer in the US, this is it:

Let’s have a look at the hourly rate of Full-Stack Web Developers next!

Hourly Rate

Full-Stack Developers also work on freelancing platforms such as Upwork or Fiverr.

• Related Article: What’s the best freelancing platform?

If you decide to go the route as a freelance Full-Stack Web Developer, you can expect to make between $35 and $60 per hour on Upwork (source). Assuming an annual workload of 2000 hours, you can expect to make between $70,000 and $120,000 per year.

Note: Do you want to create your own thriving coding business online? Feel free to check out our freelance developer course — the world’s #1 best-selling freelance developer course that specifically shows you how to succeed on Upwork and Fiverr!

Industry Demand

But is there enough demand? Let’s have a look at Google trends to find out how interest evolves over time (source):

This graphic shows that the supply of people interested in learning web development has increased since 2013.

However, if you look at the demand for web developers—it has grown even more in recent years!

As in any market, if demand outstrips supply, prices of this resource tend to increase. That’s why it can be a super lucrative decision to become a backend web developer in the 2020s, 2030s, and beyond.

Learning Path, Skills, and Education Requirements

Do you want to become a Full-Stack Web Developer? Learn both front-end and back-end skills!

Here’s a step-by-step learning path I’d propose to get started with the most crucial front-end web developments tools:

• Step 1: Introduction to HTML (~20 hours)
• Step 2: Introduction to CSS (~20 hours)
• Step 3: Introduction to JavaScript (~20 hours)
• Step 4: Angular (~20 hours)
• Step 5: Bootstrap (~20 hours)
• Step 6: React (~20 hours)

Here’s a step-by-step learning path I’d propose to get started with the most crucial back-end web developments tools and fundamental skills (master basics first! ):

• Step 1: Learn Distributed Systems (~10h)
• Step 2: Learn Databases & SQL (~10h)
• Step 3: Learn Python (~10h)
• Step 4: Learn Flask (~10h)
• Step 5: Learn JavaScript (~10h)
• Step 6: Learn Node.js (~10h)

You can find many additional computer science courses on the Finxter Computer Science Academy (flatrate model).

But don’t wait too long to acquire practical experience!

Even if you have few skills, it’s best to get started as a freelance developer and learn as you work on real projects for clients — earning income as you learn and gaining motivation through real-world feedback.

Tip: An excellent start to turbo-charge your freelancing career (earning more in less time) is our Finxter Freelancer Course. The goal of the course is to pay for itself!

Related Video

You can find more job descriptions for coders, programmers, and computer scientists in our detailed overview guide:

• Get Your Job in IT [Overview]

Related Income of Professional Developers

The following statistic shows the self-reported income from 9,649 US-based professional developers (source).

The average annual income of professional developers in the US is between $70,000 and $177,500 for various programming languages.

Question: What is your current total compensation (salary, bonuses, and perks, before taxes and deductions)? Please enter a whole number in the box below, without any punctuation. If you are paid hourly, please estimate an equivalent weekly, monthly, or yearly salary. (source)

The following statistic compares the self-reported income from 46,693 professional programmers as conducted by StackOverflow.

The average annual income of professional developers worldwide (US and non-US) is between $33,000 and $95,000 for various programming languages.

Here’s a screenshot of a more detailed overview of each programming language considered in the report:

Here’s what different database professionals earn:

Here’s an overview of different cloud solutions experts:

Here’s what professionals in web frameworks earn:

There are many other interesting frameworks—that pay well!

Look at those tools:

Okay, but what do you need to do to get there? What are the skill requirements and qualifications to make you become a professional developer in the area you desire?

Let’s find out next!

General Qualifications of Professionals

StackOverflow performs an annual survey asking professionals, coders, developers, researchers, and engineers various questions about their background and job satisfaction on their website.

Interestingly, when aggregating the data of the developers’ educational background, a good three quarters have an academic background.

Here’s the question asked by StackOverflow (source):

Which of the following best describes the highest level of formal education that you’ve completed?

However, if you don’t have a formal degree, don’t fear! Many of the respondents with degrees don’t have a degree in their field—so it may not be of much value for their coding careers anyways.

Also, about one out of four don’t have a formal degree and still succeeds in their field! You certainly don’t need a degree if you’re committed to your own success!

Freelancing vs Employment Status

The percentage of freelance developers increases steadily. The fraction of freelance developers has already reached 11.21%!

This indicates that more and more work will be done in a more flexible work environment—and fewer and fewer companies and clients want to hire inflexible talent.

Here are the stats from the StackOverflow developer survey (source):

Do you want to become a professional freelance developer and earn some money on the side or as your primary source of income?

Resource: Check out our freelance developer course—it’s the best freelance developer course in the world with the highest student success rate in the industry!

Other Programming Languages Used by Professional Developers

The StackOverflow developer survey collected 58000 responses about the following question (source):

Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?

These are the languages you want to focus on when starting out as a coder:

And don’t worry—if you feel stuck or struggle with a nasty bug. We all go through it. Here’s what SO survey respondents and professional developers do when they’re stuck:

What do you do when you get stuck on a problem? Select all that apply. (source)

Related Tutorials

To get started with some of the fundamentals and industry concepts, feel free to check out these articles:

• Freelance Developer – How to Code From Home and Earn Six Figures [Industry Report]
• How to Become a Python Freelancer—and Earn $1,000 on the Side? [A Step-by-Step Tutorial]
• How Adam Earns $5000 per Month as a Python Freelancer on Upwork [Month 4]
• Web Development Wikipedia

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

Resources

[1] We used the following code to create the income graphic:

import matplotlib.pyplot as plt
import numpy as np
import math

data = [99274,
        88133,
        108000,
        106934,
        98271,
        79584,
        108984]

labels = ['Glassdoor.com',
          'ZipRecruiter.com',
          'Kinsta.com',
          'Indeed.com',
          'Salary.com',
          'PayScale.com',
          'CareerKarma.com']

median = np.median(data)
average = np.average(data)
print(median, average)
n = len(data)

plt.plot(range(n), [median] * n, color='black', label='Median: $' + str(int(median)))
plt.plot(range(n), [average] * n, '--', color='red', label='Average: $' + str(int(average)))
plt.bar(range(len(data)), data)
plt.xticks(range(len(data)), labels, rotation='vertical', position = (0,0.45), color='white', weight='bold')
plt.ylabel('Average Income ($)')
plt.title('Full-Stack Developer Annual Income - by Finxter')
plt.legend()
plt.show()
        

You can check out our article on Matplotlib to better understand the code.

Creating Frontends in the Crypto Space

In the following video, Finxter Mikio shows how to create a front-end of a crypto-related website using Python Brownie:

If you’re interested in creating crypto front-end projects, also check out our course at the Finxter Computer Science Academy:

What Is a Back-end Web Developer?

A web developer is a programmer who specializes in the development of websites or applications viewed on web browsers, mobile devices, and large desktop screens that are transported over private or public networks such as the Internet.

A back-end web developer focuses on the implementation and deployment of the “back-end” of a web application. The back-end consists of the webserver infrastructure, databases, and code function integration to facilitate a smooth and secure serving of user requests. Back-end developers often operate behind the scenes (from a user perspective) but work closely in teams with database administrators, distributed systems experts and front-end web developers.

A front-end web developer focuses on the graphical user interface (GUI) of the website using HTML, CSS, and JavaScript with the goal of setting up the whole technology stack to enable users to view and interact with the website.

This video nicely explains some of the most important technologies and skills you need as a back-end web developer:

Who Do Back-end Web Developers Work For?

Back-end web developers either work independently as freelancers or as employees for companies, government organizations, crypto projects, or non-profits.

Lately, many back-end web developers have started to work for decentralized autonomous organizations (DAOs) in the crypto ecosystem due to their expertise in native web technologies, distributed systems, databases, and web philosophies.

In fact, the whole Blockchain technology stack is built around “back-end” web technologies—the Blockchain can be seen as a distributed, decentralized, and secure database that is replicated among thousands of participants. It’s the world of back-end web developers!

Almost all decentralized projects (including Bitcoin and Ethereum) only focus on decentralized back-end development whereas the front-ends run on a centralized infrastructure!

Now that you know about what it is, let’s have a look at what it earns next!

Annual Income

How much does a Back-end Web Developer make per year?

The average annual income of a Back-End Web Developer in the United States is between $72,484 and $140,000 with an average income of $96,545 and a median income of $97,522 per year according to our meta-study of 9 aggregated data sources such as Glassdoor and Indeed.

The following graphic shows the individual data sources, as well as the average and median income level of a back-end web developer in the US:

Interestingly, there is a statistically significant difference in both the median and the average income of a back-end web developer versus a front-end web developer and a general web developer.

• Back-end web developers make on average $96,545 (median: $97,522).
• Front-end web developers make on average $89,683 (median: $90,499) per year.
• General web developers make on average $88,054 (median: $90,000) per year.

Here’s the income of a front-end web developer for comparison:

Here’s the income of a general web developer for comparison:

If you need the raw data for the income of a back-end web developer in the US, this is it:

Let’s have a look at the hourly rate of Back-end Web Developers next!

Hourly Rate

Back-end Web Developers also work on freelancing platforms such as Upwork or Fiverr.

• Related Article: What’s the best freelancing platform?

If you decide to go the route as a freelance Back-End Web Developer, you can expect to make between $25 and $50 per hour on Upwork (source). Assuming an annual workload of 2000 hours, you can expect to make between $50,000 and $100,000 per year.

This is one of the rare cases where it may be better to work as an employee compared to working as a freelancer. The reason is that only few companies would give external freelancers the keys to their “back-end” kingdoms!

It is much easier for a business to hire an external freelancer for front-end user interfaces (CSS, HTML, JavaScript) than to hire a freelancer to handle proprietary and privacy-sensitive customer data, for instance.

Note: Do you want to create your own thriving coding business online? Feel free to check out our freelance developer course — the world’s #1 best-selling freelance developer course that specifically shows you how to succeed on Upwork and Fiverr!

Industry Demand

But is there enough demand? Let’s have a look at Google trends to find out how interest evolves over time (source):

This graphic shows that the supply of people interested in learning web development has increased since 2013.

However, if you look at the demand for web developers—it has exploded in recent years!

As in any market, if demand outstrips supply, prices of this resource tend to increase. That’s why it can be a super lucrative decision to become a backend web developer in the 2020s, 2030s, and beyond.

Learning Path, Skills, and Education Requirements

Do you want to become a Back-end Web Developer?

Here’s a step-by-step learning path I’d propose to get started with the most crucial back-end web developments tools and fundamental skills (master basics first! ):

• Step 1: Learn Distributed Systems (~10h)
• Step 2: Learn Databases & SQL (~10h)
• Step 3: Learn Python (~10h)
• Step 4: Learn Flask (~10h)
• Step 5: Learn JavaScript (~10h)
• Step 6: Learn Node.js (~10h)

You can find many additional computer science courses on the Finxter Computer Science Academy (flatrate model).

But don’t wait too long to acquire practical experience!

Even if you have few skills, it’s best to get started as a freelance developer and learn as you work on real projects for clients — earning income as you learn and gaining motivation through real-world feedback.

Tip: An excellent start to turbo-charge your freelancing career (earning more in less time) is our Finxter Freelancer Course. The goal of the course is to pay for itself!

Related Video

You can find more job descriptions for coders, programmers, and computer scientists in our detailed overview guide:

• Get Your Job in IT [Overview]

Web Developer Comparisons

web developer vs web designer

A web developer creates the core functionality of a website whereas a web designer is a graphic artist responsible for designing the layout, usability, and visual appearance of a website. A successful web designer often has outstanding skills in creativity, graphic design, and technical understanding.

• The average income of a web developer in the US is $88,054 per year.
• The average income of a web designer in the US is $60,000 per year.

web developer vs front-end developer

A web developer creates the core functionality of a website whereas a front-end developer is concerned with the functionality of the user interface on the browser. Compared to a web designer, a front-end developer is more concerned with the functionality and user experience (e.g., implementing buttons and user inputs functionality rather than designing them).

• The average income of a web developer in the US is $88,054 per year.
• The average income of a front-end developer in the US is $89,683 per year.

web developer vs software developer (programmer, software engineer)

A web developer specializes in web applications such as websites, e-commerce, and mobile apps, whereas a software developer (engineer) specializes in creating software for the underlying operating system, network, or platform.

All web developers are software developers but not all software developers are web developers!

• The average income of a web developer in the US is $88,054 per year.
• The average income of a software developer in the US is $110,140 per year.

web developer vs data analyst

Web developers create websites and web apps for companies whereas data scientists (data analysts) draw insights from structured and unstructured data using a multitude of tools such as machine learning, visualization, and statistical analysis.

• The average income of a web developer in the US is $88,054 per year.
• The average income of a data scientist in the US is $122,700 per year.

Related Income of Professional Developers

The following statistic shows the self-reported income from 9,649 US-based professional developers (source).

The average annual income of professional developers in the US is between $70,000 and $177,500 for various programming languages.

Question: What is your current total compensation (salary, bonuses, and perks, before taxes and deductions)? Please enter a whole number in the box below, without any punctuation. If you are paid hourly, please estimate an equivalent weekly, monthly, or yearly salary. (source)

The following statistic compares the self-reported income from 46,693 professional programmers as conducted by StackOverflow.

The average annual income of professional developers worldwide (US and non-US) is between $33,000 and $95,000 for various programming languages.

Here’s a screenshot of a more detailed overview of each programming language considered in the report:

Here’s what different database professionals earn:

Here’s an overview of different cloud solutions experts:

Here’s what professionals in web frameworks earn:

There are many other interesting frameworks—that pay well!

Look at those tools:

Okay, but what do you need to do to get there? What are the skill requirements and qualifications to make you become a professional developer in the area you desire?

Let’s find out next!

General Qualifications of Professionals

StackOverflow performs an annual survey asking professionals, coders, developers, researchers, and engineers various questions about their background and job satisfaction on their website.

Interestingly, when aggregating the data of the developers’ educational background, a good three quarters have an academic background.

Here’s the question asked by StackOverflow (source):

Which of the following best describes the highest level of formal education that you’ve completed?

However, if you don’t have a formal degree, don’t fear! Many of the respondents with degrees don’t have a degree in their field—so it may not be of much value for their coding careers anyways.

Also, about one out of four don’t have a formal degree and still succeeds in their field! You certainly don’t need a degree if you’re committed to your own success!

Freelancing vs Employment Status

The percentage of freelance developers increases steadily. The fraction of freelance developers has already reached 11.21%!

This indicates that more and more work will be done in a more flexible work environment—and fewer and fewer companies and clients want to hire inflexible talent.

Here are the stats from the StackOverflow developer survey (source):

Do you want to become a professional freelance developer and earn some money on the side or as your primary source of income?

Resource: Check out our freelance developer course—it’s the best freelance developer course in the world with the highest student success rate in the industry!

Other Programming Languages Used by Professional Developers

The StackOverflow developer survey collected 58000 responses about the following question (source):

Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?

These are the languages you want to focus on when starting out as a coder:

And don’t worry—if you feel stuck or struggle with a nasty bug. We all go through it. Here’s what SO survey respondents and professional developers do when they’re stuck:

What do you do when you get stuck on a problem? Select all that apply. (source)

Related Tutorials

To get started with some of the fundamentals and industry concepts, feel free to check out these articles:

• Freelance Developer – How to Code From Home and Earn Six Figures [Industry Report]
• How to Become a Python Freelancer—and Earn $1,000 on the Side? [A Step-by-Step Tutorial]
• How Adam Earns $5000 per Month as a Python Freelancer on Upwork [Month 4]
• Web Development Wikipedia

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!

Resources

[1] We used the following code to create the income graphic:

import matplotlib.pyplot as plt
import numpy as np
import math

data = [94455,
        83770,
        80713,
        97522,
        102958,
        110000,
        87009,
        140000,
        72484]

labels = ['Glassdoor.com',
          'ZipRecruiter.com',
          'Kinsta.com',
          'Indeed.com',
          'Salary.com',
          'Comparably.com',
          'PayScale.com',
          'Talent.com',
          'Arc.dev']

median = sorted(data)[math.ceil(len(data)/2)]
average = sum(data)/len(data)
print(median, average)
n = len(data)

plt.plot(range(n), [median] * n, color='black', label='Median: $' + str(int(median)))
plt.plot(range(n), [average] * n, '--', color='red', label='Average: $' + str(int(average)))
plt.bar(range(len(data)), data)
plt.xticks(range(len(data)), labels, rotation='vertical', position = (0,0.45), color='white', weight='bold')
plt.ylabel('Average Income ($)')
plt.title('Back Web Developer Annual Income - by Finxter')
plt.legend()
plt.show()
        

You can check out our article on Matplotlib to better understand the code.

Creating Frontends in the Crypto Space

In the following video, Finxter Mikio shows how to create a front-end of a crypto-related website using Python Brownie:

If you’re interested in creating crypto front-end projects, also check out our course at the Finxter Computer Science Academy:

How much does a RavenDB Developer make per year?

The average annual income of a RavenDB Developer is hard to estimate given the limited data availability. However, RavenDB developers are NoSQL developers who earn between $101,000 (25th percentile) and $130,500 (75th percentile) with an average of $119,105 per year in the US according to Ziprecruiter (source).

Top earners focusing on RavenDB development can make $155,500 and more!

Let’s have a look at the hourly rate of RavenDB Developers next!

Hourly Rate

RavenDB Developers are well-paid on freelancing platforms such as Upwork or Fiverr.

• Related Article: What’s the best freelancing platform?

If you decide to go the route as a freelance RavenDB Developer, you can expect to make between $20 and $60 per hour on Upwork (source). Assuming an annual workload of 2000 hours, you can expect to make between $40,000 and $120,000 per year.

Note: Do you want to create your own thriving coding business online? Feel free to check out our freelance developer course — the world’s #1 best-selling freelance developer course that specifically shows you how to succeed on Upwork and Fiverr!

Industry Demand

But is there enough demand? Let’s have a look at Google trends to find out how interest evolves over time (source):

Work Description

So, you may wonder: RavenDB Developer – what’s the definition?

RavenDB Developer Definition: A RavenDB Developer developer creates, edits, analyzes, debugs, and supervises the development of software written in the RavenDB programming language. RavenDB is a NoSQL document-oriented database written especially for the .NET framework. (Source)

Learning Path, Skills, and Education Requirements

Do you want to become a RavenDB Developer? Here’s a step-by-step learning path I’d propose to get started with RavenDB:

• Step 1: Introduction to Computer Science (~40 hours)
• Step 2: Introduction to Algorithms (~40 hours)
• Step 3: Introduction to RavenDB (~20 hours)
• Step 4: Introduction to RavenDB (Free Video Courses) (~10 hours)

You can find many additional computer science courses on the Finxter Computer Science Academy (flatrate model).

But don’t wait too long to acquire practical experience!

Even if you have little skills, it’s best to get started as a freelance developer and learn as you work on real projects for clients — earning income as you learn and gaining motivation through real-world feedback.

Tip: An excellent start to turbo-charge your freelancing career (earning more in less time) is our Finxter Freelancer Course. The goal of the course is to pay for itself!

Related Video

You can find more job descriptions for coders, programmers, and computer scientists in our detailed overview guide:

• Get Your Job in IT [Overview]

Related Income of Professional Developers

The following statistic shows the self-reported income from 9,649 US-based professional developers (source).

The average annual income of professional developers in the US is between $70,000 and $177,500 for various programming languages.

Question: What is your current total compensation (salary, bonuses, and perks, before taxes and deductions)? Please enter a whole number in the box below, without any punctuation. If you are paid hourly, please estimate an equivalent weekly, monthly, or yearly salary. (source)

The following statistic compares the self-reported income from 46,693 professional programmers as conducted by StackOverflow.

The average annual income of professional developers worldwide (US and non-US) is between $33,000 and $95,000 for various programming languages.

Here’s a screenshot of a more detailed overview of each programming language considered in the report:

Here’s what different database professionals earn:

Here’s an overview of different cloud solutions experts:

Here’s what professionals in web frameworks earn:

There are many other interesting frameworks—that pay well!

Look at those tools:

Okay, but what do you need to do to get there? What are the skill requirements and qualifications to make you become a professional developer in the area you desire?

Let’s find out next!

General Qualifications of Professionals

StackOverflow performs an annual survey asking professionals, coders, developers, researchers, and engineers various questions about their background and job satisfaction on their website.

Interestingly, when aggregating the data of the developers’ educational background, a good three quarters have an academic background.

Here’s the question asked by StackOverflow (source):

Which of the following best describes the highest level of formal education that you’ve completed?

However, if you don’t have a formal degree, don’t fear! Many of the respondents with degrees don’t have a degree in their field—so it may not be of much value for their coding careers anyways.

Also, about one out of four don’t have a formal degree and still succeeds in their field! You certainly don’t need a degree if you’re committed to your own success!

Freelancing vs Employment Status

The percentage of freelance developers increases steadily. The fraction of freelance developers has already reached 11.21%!

This indicates that more and more work will be done in a more flexible work environment—and fewer and fewer companies and clients want to hire inflexible talent.

Here are the stats from the StackOverflow developer survey (source):

Do you want to become a professional freelance developer and earn some money on the side or as your primary source of income?

Resource: Check out our freelance developer course—it’s the best freelance developer course in the world with the highest student success rate in the industry!

Other Programming Languages Used by Professional Developers

The StackOverflow developer survey collected 58000 responses about the following question (source):

Which programming, scripting, and markup languages have you done extensive development work in over the past year, and which do you want to work in over the next year?

These are the languages you want to focus on when starting out as a coder:

And don’t worry—if you feel stuck or struggle with a nasty bug. We all go through it. Here’s what SO survey respondents and professional developers do when they’re stuck:

What do you do when you get stuck on a problem? Select all that apply. (source)

Related Tutorials

To get started with some of the fundamentals and industry concepts, feel free to check out these articles:

• Freelance Developer – How to Code From Home and Earn Six Figures [Industry Report]
• How to Become a Python Freelancer—and Earn $1,000 on the Side? [A Step-by-Step Tutorial]
• How Adam Earns $5000 per Month as a Python Freelancer on Upwork [Month 4]
• RavenDB Wikipedia
• Learn RavenDB Google

Where to Go From Here?

Enough theory. Let’s get some practice!

Coders get paid six figures and more because they can solve problems more effectively using machine intelligence and automation.

To become more successful in coding, solve more real problems for real people. That’s how you polish the skills you really need in practice. After all, what’s the use of learning theory that nobody ever needs?

You build high-value coding skills by working on practical coding projects!

Do you want to stop learning with toy projects and focus on practical code projects that earn you money and solve real problems for people?

If your answer is YES!, consider becoming a Python freelance developer! It’s the best way of approaching the task of improving your Python skills—even if you are a complete beginner.

If you just want to learn about the freelancing opportunity, feel free to watch my free webinar “How to Build Your High-Income Skill Python” and learn how I grew my coding business online and how you can, too—from the comfort of your own home.

Join the free webinar now!