According to Python’s founder Guido van Rossum, Python provides “Computer Programming for Everybody”. Coders love Python for several reasons. Its smooth learning curve, readability, comprehensibility of the Python Standard Library, and fantastic 3rd party frameworks. Data science (manipulation and visualization), (Web) Scraping, Machine Learning, Artificial Intelligence, Game Development, Web Development are some of the wide range areas you can achieve with Python and its libraries.
I have recently experienced the case while launching my latest web application project: Fazla.net facts and stats. I am the solopreneur/solo developer of the full-stack web project.
The process of making the project online had 4 main steps:
- Web scraping to gather data,
- Data manipulation to make it meaningful for my database,
- Web application itself
- Deployment
All the steps are achieved with Python code.
I have used beautifulsoup and requests libraries for the first step, web scraping. They make it very easy to gather data from a website. For small tasks like mine, there is even no learning curve. A few lines of code do the job very well and satisfying.
Step 1: Getting the Data
The datasets in your harddisk are nothing more than luggage taken from the home. The road starts here and you should go far with that luggage.
Step 2: Manipulating the Data
The second step, data manipulation, is so important because your way of modeling the database generally differs from the source’s spreadsheets or APIs. Hundreds of thousands of rows of data can be intimidating and a forever lasting task if you try to do it on excel itself. For this job, two python libraries are life-saving; NumPy and pandas.
As of the time of writing this post, there are about 100k rows of data served by fazla.net and I have manipulated roughly half a million rows of data which will be online in the future versions of the application. For data scientific works, I have written more than a few thousands of lines of code. pd.read_excel … df.to_excel code script was used for many different excel sheets.
Step 3: Creating the Web App with Django
The third step was achieved with batteries included python framework: Django. I am a freelance full-stack Django developer getting paid for.
My last two years have passed with achieving Django related tasks and finding solutions to the issues. The backend Django part of the app was the second easier part of the four steps. In my opinion, as mentioned by seasoned web developers all the time, coding the models with their relationships is the most important part of a Django application. You can do everything later, but if your models are stuck at some time, their relationships will make it hard to maintain.
On the other hand, models, in a way, plot your app’s route. Views, templates, APIs, and so on are all affected by the models. Django has its own many useful third-party libraries which help a lot. Front-end is another sub-step of establishing a Django application. Knowledge of HTML, CSS, and JS is a must, even if you are using 3rd party templates.
As a backend loving developer, I don’t feel comfortable with front-end tasks and technologies, but I push hard to work professionally. Being a statistics website, the application has intensive use of charts and data tables. Defining the right one from the alternatives outside was a hard task. One chart library is fast, but cannot provide a charming UI. Another one is great, but you cannot use it on a public page because it seems to show up a year later. I have played with several of them and I think the choosing process will continue because I am not fully satisfied with the current situation. An online application should have a quick response time to visitors’ requests.
Step 4: Deployment
The fourth step is deploying the application. Choosing the right cloud service provider, affording the expenses, the configuration of the server, security, and making the app fast may fit here. To make the application load fast, you have two options. One is to choose several hundred bucks per month costing a cloud host or making the app’s optimizations/using CDN. I have done some optimizations including reducing DB queries (run away from n+1 issue), compression and reducing the number of static files, using CDN, and so on. It is not over yet, there is much more to do. My target is to reduce the full loading time of a page under one second.
Lastly, there may be more steps not mentioned here. Those were the ones I have seen the most important. By the way, before all of them, learning python is an important task to be accomplished well. I am not a seasoned Python developer, but I have learned much from Christian, his finxter app, and his books. They deserve much to be mentioned here to accomplish a stable python background.
About The Author
Haluk Aksu: I am a full-stack Python/Django developer. There are some freelance private projects I have done. Also, there are some open-source projects on my GitHub account. My latest project is Fazla.net facts and stats with the motto ‘stats are one click away’.