Subtitle: Avoid Hard-Coded Data While Making Your (And Others) Life Easier
When the subject of config, or configuration, files comes up there is often confusion on what they are, how to read them, and the benefits that come from their use. In this article, we will cover what config files are before introducing the Python module configparser.py. We’ll then use some examples to help you to understand, create, read, and update config files of your own.
A Quick Introduction To Configuration Files
While primarily used by operating systems to customise an environment, config files may be used for a broad set of uses. The most common use cases for coders or developers is to configure initial settings for a programme without hard coding data into the programme itself. This allows you to change those settings without the need to enter the programme and edit the source code itself. So in short, those things that may change over time such as IP addresses, host-names, user names, passwords, database names, can all be structured into a config file and the programme directed to that file for the specific information required. For those who wish to provide public access to their code on a repository like GitHub, config files may be included in the .gitignore file to ensure they remain inaccessible to the public, thereby protecting sensitive information.
How Are They Structured?
I am not aware of any standards or conventions concerning the layout of config files. Several formats allow quite complex data structures to be easily stored for later recovery and use. The deciding factor on the format you use will come down to what your programming language most easily handles and your own personal preference. Given our interest in Python, I’ll concentrate on one common format but really, the sky is the limit on what you can use. Common formats include INI, JSON, YAML and XML. Today we’ll focus on the INI format.
Creating, Reading, And Updating A config.ini File
Back in the day, Microsoft Windows relied largely on initialisation files, or INI files, to configure operating system and applications settings. In Python, we have a module called configparser.py which allows us to easily create, read and update INI files. If you don’t already have it installed, use the command
$ pip install configparser
Structure of an INI file
Each INI file you create will consist of sections within which data is stored using key-value pairs in much the same way as a Python dictionary datatype. The following is a handwritten sample of an INI file which we will create, then we will read from it, before finally updating it:
Planned Filename: configfile.ini
[postgresql] host = localhost user = finxter1 password = myfinxterpw db = postgres [user_info] admin = David Yeoman login = finxter_freelancer password = freelancer_freedom
Note that this proposed INI file consists of two sections, ‘postgresql’ and ‘user_info’. Under each section is the data keys (host, user, password etc.) and the data values (localhost, finxter1, myfinxterpw etc.).
So this configfile.ini might be useful if our code needs to create an SQL query to our Postgres database, or if we need to sign in to a particular site requiring admin privileges, login, and password data.
You can add as many sections as you wish for multiple use cases.
Creating a new config file
Open a new Python file in which we will write the code to create the INI file. The following code is largely self-explanatory. Note that in this code I’m assuming no file of the same name currently exists hence the ‘w’ command for ‘write’ in the second to last line of the code. Here are the steps
- Import the
configparsermodule - Create the
configparserobject and pass it to‘config’ - Create a section name.
- Populate the section with key: value pairs (such as
‘host’ = ‘local_host’in our example) - Repeat items 3 and 4 for any subsequent sections.
- Finally, open a file of your chosen name at your chosen directory location using
‘w’to write - Write the file
So here is the code that follows the steps above.
import configparser
config = configparser.ConfigParser()
# Add the structure to the file we will create
config.add_section('postgresql')
config.set('postgresql', 'host', 'localhost')
config.set('postgresql', 'user', 'finxter1')
config.set('postgresql', 'port', '5543')
config.set('postgresql', 'password', 'myfinxterpw')
config.set('postgresql', 'db', 'postgres')
config.add_section('user_info')
config.set('user_info', 'admin', 'David Yeoman')
config.set('user_info', 'login', 'finxter_freelancer')
config.set('user_info', 'password', 'freelancer_freedom')
# Write the new structure to the new file
with open(r"C:\PythonTutorials\configfile.ini", 'w') as configfile:
config.write(configfile)
Execute the file and check the directory you entered in the code to find the new file. Here is a screenshot of the file I’ve created with the code above.
Congratulations. You’ve written your first INI file. Now, how do we use it?
Reading a config file
For the sake of this exercise, let us assume you wish to create an SQL query from some new source code you’re writing. How would we read the config file and use the data? For those of you who aren’t into SQL let’s do a quick run-through of how you would structure this without a config file, then we’ll show you how to structure it with one. It will be easy to see the difference. Here are the steps without a config file.
- Import SQLAlchemy (my preferred tool for Postgresql queries)
- You next need to set all the parameters that will form a URI to allow connection to the database
- Create your URI and your connection engine and pass it to a variable
- Connect to your database
Here is the code you’d use without using a config file.
import sqlalchemy
# set your parameters for the database connection URI
user = "finxter1"
password = "myfinxterpw"
host = "localhost"
port = 5543
dbase = "postgres"
# Create the URI and the engine with the URI in the following format
# postgresql://username:password@hostname:port/database
# therefore we pass the URI the parameters established above
conn = sqlalchemy.create_engine('postgresql://user:password@host:port/dbase', encoding='utf8')
# Connect to the database
connection = conn.raw_connection()
So when we execute this code we’re now connected to our database, but there’s an obvious problem. We have hard-coded our username and password into the source code for any and all to see. Not clever.
Now let’s use this same code with some additions to allow us to use the configfile.ini that we created above. We have two extra steps, import configparser and then read the configfile.ini. Here are the steps
- Import SQLAlchemy
- Import configparser
- Read the configfile.ini
- Read the parameters from the configfile.ini that will form a URI to connect to the database
- Create your URI and your connection engine and pass it to a variable
- Connect to your database
Here is the code you’d use when connecting using your INI file.
import sqlalchemy
import configparser
#Read config.ini file
config_obj = configparser.ConfigParser()
config_obj.read("C:\PythonTutorials\configfile.ini")
dbparam = config_obj["postgresql"]
useinfo = config_obj["user_info"]
# set your parameters for the database connection URI using the keys from the configfile.ini
user = dbparam["user"]
password = dbparam["password"]
host = dbparam["host"]
port = int(dbparam["port"])
dbase = dbparam["db"]
# Create the URI and the engine with the URI in the following format
# postgresql://username:password@hostname:port/database
# so we pass the URI the parameters established above
conn = sqlalchemy.create_engine('postgresql://user:password@host:port/dbase', encoding='utf8')
# Connect to the database
connection = conn.raw_connection()As you can see we needed to use configparser to create an object, then read the configfile.ini that we created earlier. Having read the configfile.ini we passed each section of the file to separate variables, in this case, the ‘postgresql’ parameters to dbparam and the ‘user_info’ parameters to useinfo.
#Read config.ini file
config_obj = configparser.ConfigParser()
config_obj.read("C:\PythonTutorials\configfile.ini")
dbparam = config_obj["postgresql"]
useinfo = config_obj["user_info"]
So instead of hard coding the values into the parameters for the database connection as we did previously, we can now call each value we need from the configfile.ini by using the keys and assigning them to variables.
# set your parameters for the database connection URI using the keys from the configfile.ini user = dbparam["user"] password = dbparam["password"] host = dbparam["host"] port = int(dbparam["port"]) dbase = dbparam["db"]
To prove that the data has been appropriately transferred, let’s use a print command to check.
import sqlalchemy
import configparser
#Read config.ini file
config_obj = configparser.ConfigParser()
config_obj.read("C:\PythonTutorials\configfile.ini")
dbparam = config_obj["postgresql"]
useinfo = config_obj["user_info"]
# set your parameters for the database connection URI using the keys from the configfile.ini
user = dbparam["user"]
password = dbparam["password"]
host = dbparam["host"]
port = int(dbparam["port"])
dbase = dbparam["db"]
print('User variable = ', user, '\n')
print('Password variable = ', password, '\n')
print('Host variable = ', host, '\n')
print('Port variable = ', port, '\n')
print('Database variable = ', dbase, '\n')
# Result
User variable = finxter1
Password variable = myfinxterpw
Host variable = localhost
Port variable = 5543
Database variable = postgres
Process finished with exit code 0
So admittedly a bit more code in the second example, but look at the benefits you get.
- You don’t need to hard code sensitive information into a file that others can explore
- No matter how many times you use the parameters throughout your code, if you need to change their values you simply edit the config file
- If you wish to post your code on a repository like GitHub you can place the config file in the
.gitignorefile to be sure others cannot access your secret information, then you add a readme file to instruct others how to create a new config file with their own information
Updating your config file
This is simplicity itself. Here are two options.
The first is to open the configfile.ini in notepad and manually make the changes. Let’s change the ‘user’ details from finxter1 to coding_finxter8765.
Now run the code again.
import sqlalchemy
import configparser
#Read config.ini file
config_obj = configparser.ConfigParser()
config_obj.read("C:\PythonTutorials\configfile.ini")
dbparam = config_obj["postgresql"]
useinfo = config_obj["user_info"]
# set your parameters for the database connection URI using the keys from the configfile.ini
user = dbparam["user"]
password = dbparam["password"]
host = dbparam["host"]
port = int(dbparam["port"])
dbase = dbparam["db"]
print('User variable = ', user, '\n')
print('Password variable = ', password, '\n')
print('Host variable = ', host, '\n')
print('Port variable = ', port, '\n')
print('Database variable = ', dbase, '\n')
# Result
User variable = coding_finxter8765
Password variable = myfinxterpw
Host variable = localhost
Port variable = 5543
Database variable = postgres
The second option is to use configparser from within Python. Let’s change the PostgreSQL password from myfinxterpw to finxter_coders_rock. Here’s the code.
import configparser
# Read config.ini file
edit = configparser.ConfigParser()
edit.read("C:/PythonTutorials/configfile.ini")
#Get the postgresql section
postgresql = edit["postgresql"]
#Update the password
postgresql["password"] = "finxter_coders_rock"
#Write changes back to file
with open('C:/PythonTutorials/configfile.ini', 'w') as configfile:
edit.write(configfile)
Here’s the result.
To Summarise
Today we discussed configuration or ‘config’ files. What they are, why they’re used and some common formats used to structure them.
We introduced the configparser module used in Python and wrote a script to create a new INI file called configfile.ini.
We then illustrated the use (and perils) of hard-coded values when creating an SQL connection, before amending the SQL connection script to allow us to access the necessary values from the configfile.ini which we had created.
Finally, we demonstrated how easy it is to amend an INI file by manually amending some data with a word processor before we returned to the configparser module to make further amendments.
Remember – I have used an example here of creating a database connection, but you can use a config file for any data that you wish to remain secure, or for data that is often used throughout some source code and which may change from time to time. Any changes may be made once by editing the config file rather than sorting through hundreds of lines of code and manually editing all occurrences.
I hope this article has been of use to you. Thanks for reading.