Dockerized PostgreSQL and Django for Local Development

Docker and docker-compose make it dead simple to avoid dependency hell and have a consistent environment for your whole team while doing local development. This post walks through setting up a new Django project from scratch to use Docker and docker-compose. It is modeled after a previous post that I wrote about doing a similar thing with Laravel and MySQL.

Dockerfile

Nothing too interesting happening here. Installing python and pip.

FROM ubuntu:16.04

# system update
RUN apt update
RUN apt upgrade -y

# python deps
RUN apt install -y python3-dev python3-pip

docker-compose.yml

version: '2'
services:
  app:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    working_dir: /app
    command: bash -c "pip3 install -r requirements.txt && python3 manage.py migrate && python3 manage.py runserver 0:8000"
    depends_on:
      - db
  db:
    image: postgres:9.6.5-alpine
    environment:
      - POSTGRES_USER=feedread
      - POSTGRES_PASSWORD=feedread
    volumes:
      - ./data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

With this in place you can start your Django app with docker-compose up. Each time the app starts it will install the latest dependencies, run migrations, and start serving the app on localhost:8000

Notes

  1. In order to do stuff with the database locally you should add the following record to your local /etc/hosts file
    # /etc/hosts
    
    127.0.0.1 db
    
  2. Since we define – .:/app as a volume, this means that all of your local changes are immediately visible in the dockerized app.
  3. If you need to access the running app or db container you can do so with docker-compose exec app bash or docker-compose exec db bash.
  4. This docker-compose file is not really suitable for production since it is not likely that you would want to build the container each time the app starts or automatically run migrations.
  5. You can add additional services like memcached, a mail server, an app server, a queue, etc., using the same method that we are using above with our database.

I Want to Become a Core Python Developer

I’ve been tinkering with python for almost five years now. I am absolutely in love with the language. My new goal is to make enough contributions to the project to join the core team.

This post is my attempt to keep a list of all that I’ve done in this endeavor. I will keep this up to date on a monthly basis.

Short Term Goals

  • Ship some actual code. Focus on improving test coverage.
  • Attend the next available local meetup.
  • Get this PR merged.
  • Work on some other low hanging fruit from bedevere.

November 2017

Code

  • Reported an issue with Vagrant and Ansible on the pythondotorg repo, and assisted with testing the resolution. (note, for any future newbies, reporting issues, writing docs, testing PRs, these are all super valuable things that you can do to get more familiar with a projects code base).
  • Substantial refactoring of the dev guide merged.

Community

  • Reached out to the core workflow team to see if we could introduce CircleCI into the Python organization. This addresses the PoC showed in this PR.

October 2017

Code

Community

  • Became a PSF Member.
  • Hang out in various IRC channels. Notably #python on freenode and help out where I can.
  • Join the PSF Volunteers mailing list and volunteer for opportunities as they come in.
  • Sign up for all the dev related mailing lists.
  • Joined the BAyPIGgies local python meetup group.

Python Mocks Test Helpers

I’ve been writing a python wrapper for the CircleCI API over the last week. I wanted to do this “the right way” with test driven development.

I have a couple integration tests that actually hit the CircleCI API, but most of the unit tests so far are using MagicMock to ensure that the basic functions are working as expected.

This generally involves the tedious process of dumping out JSON, saving it to a file, and then reloading that file later on to actually test it.

I wrote two helper functions that make this process slightly less tedious.

Load Mock

The first is a function that loads a file and overrides every request to return that file (typically as JSON).

    def loadMock(self, filename):
        """helper function to open mock responses"""
        filename = 'tests/mocks/{0}'.format(filename)

        with open(filename, 'r') as f:
            self.c._request = MagicMock(return_value=f.read())

Test Helper

The second is a function that runs a real request for the first time and dumps the output to a file.

    def test_helper(self):
        resp = self.c.add_circle_key()
        print(resp)
        with open('tests/mocks/mock_add_circle_key_response', 'w') as f:
             json.dump(resp, f)

Naming it test_helper allows it to be picked up and ran when you run your test suite since by default unittest will capture any methods that start with test.

Usage

An actual example is shown below.

    def test_clear_cache(self):
        self.loadMock('mock_clear_cache_response')
        resp = json.loads(self.c.clear_cache('levlaz', 'circleci-sandbox'))

        self.assertEqual('build dependency caches deleted', resp['status'])

Writing the tests is easy, we just copy and paste the name of the file that was created with test_helper and verify that the contents are what we expect them to be.

This approach has been working very well for me so far. One thing to keep in mind with writing these types of tests is that you should also include some general integration tests against the API that you are working with. This way you can catch any regressions with your library in the event that the API changes in any way. However, as a basic sanity check mocking these requests is a good practice and less prone to flakiness.

Spring Security, Webjars, and MIME type error

I volunteered to be JLO (Java Language Owner) at CircleCI and I am currently working on getting a sample Spring Framework project running on CircleCI 2.0. I made a simple app bootstrapped with the Spring Initializer. I included Spring Security for the first time and I decided to try out WebJars for static Javascript libraries such as bootstrap. I am using Thymeleaf for templating.The app does not actually do anything yet but I ran into a pretty strange issue today that I wanted to write up here. My home page is pretty straightforward.

<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org">
<head>
    <title>CircleCI Spring Demo</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

    <link rel="stylesheet" th:href="@{/webjars/bootstrap/3.3.7/css/bootstrap.min.css}" />

    <link rel="stylesheet" th:href="@{/css/style.css}" href="../static/css/style.css" />
</head>
<body>

    <nav class="navbar">
        <div class="container">
            <div class="navbar-header">
                <a class="navbar-brand" href="#">CircleCI Demo Spring</a>
            </div>
            <div id="navbar" class="collapse navbar-collapse">
                <ul class="nav navbar-nav">
                    <li class="active"><a href="#">Home</a></li>
                    <li><a href="#">About</a></li>
                </ul>
            </div>
        </div>
    </nav>

    <div class="container">
        <h1> CircleCI Spring Demo </h1>
    </div>

    <script th:src="@{/webjars/bootstrap/3.3.7/js/bootstrap.min.js}"></script>
</body>
</html>

However, when I tried to load up the app with mvn spring-boot:run none of the styles showed up and console showed the following error message:

Resource interpreted as Stylesheet but transferred with MIME type text/html

It turns out, that a default spring-security config will basically block any request unless you whitelist it. The MIME type is a red herring since what is actually happening is that my spring-security config is redirecting all unauthenticated users to my login page (which is login.html) instead of serving up the stylesheet from the/webjars directory. The solution is to update my security configuration to whitelist anything that comes from /webjars

package com.circleci.demojavaspring;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;

@Configuration
@EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
            .authorizeRequests()
                .antMatchers("/", "/home", "/webjars/**").permitAll()
                .anyRequest().authenticated()
                .and()
            .formLogin()
                .loginPage("/login")
                .permitAll()
                .and()
            .logout()
                .permitAll();
    }
}

Now, the styles load as expected.

Using gtk-doc with Anjuta on Debian Stable

gtk-doc is a library that helps extract code documentation. When you create a new project with Anjuta it asks if you wish to include gkt-doc. Unfortunately, on Debian stable there seems to be a bug because the autoconf configuration is looking for the wrong version of gtk-doc.

/home/levlaz/git/librefocus/configure: line 13072: syntax error near unexpected token `1.0'
/home/levlaz/git/librefocus/configure: line 13072: `GTK_DOC_CHECK(1.0)'

On Debian stable, the version of GTK doc that comes with thegtk-doc-tools package is 1.21. In order to resolve this error you need to update configure.ac to use the newer version of gtk-doc as shown below:

GTK_DOC_CHECK([1.21])

Then you need to regenerate the entire project and everything should work as expected.

Do not Install Karma Globally

Wow, I spent so long trying to figure out why the hell karma was not working for me, it turns out its because it was installed globally. For instance. In my projects package.json I had:

"scripts": { "test": "karma start karma.conf.js" } ...

When I ran npm test – it told me sh 1: karma not found Every other possible combination also did the same thing. i.e.

node_modules/karma/bin/karma

./node_modules/karma/bin/karma

node ./node_modules/karma/bin/karma

I could totally execute this myself from the shell, so I had no idea what was wrong. Then I finally stumbled upon this GitHub Issue. After uninstalling karma globally, npm uninstall -g karma I was able to run npm test without any issues. I still have no idea why this works or didn’t work. But at this point I just want to go back to writing tests.

Injecting Stuff into your Python Path

Similar to a previous post where I wrote about how to run flask tests without installing your app, another common thing that you might want to be able to do is import your app from some arbitrary script. This is especially useful when running your app with apache mod_wsgi. This module expects the app to be installed globally or at least in the python path. Unless you install the app in a traditional sense this will not be true. The solution is just to inject the path prior to running your import statement like this.

sys.path.insert(0, '/var/www/blog')
from blog import app as application

This import will actually work.

Using the Flask CLI

Who knew that flask had a cli? Previously I used to just use manage.py just like Django does it to “do stuff”. The CLI is great, but again it follows the theme of kind of wanting you to install your flask app. (I really should do this). So in order to get your app to work you must point the FLASK_APP variable to the actual python file (not your app module). This is true even if you have a true python module. For instance.

export FLASK_APP = blog/blog.py
flask run

Works, while

export FLASK_APP = blog
flask run

Does not. Even though blog consists of:

blog/
  __init__.py
  blog.py

Follow Me on Twitter CTA on Ghost and WordPress

The other day I read a post on Hacker News about premature optimization in web application development. This was an excellent post in its own right, but one thing that jumped out at me was the call to action from the author to follow him on Twitter at the end of his post. I liked that it was subtle, simple, and effective  (I followed him on Twitter). Naturally, I stole this idea for my own blogs. This method uses the Follow Button provided by Twitter. In this post, I will show you how to add your own “Follow Me on Twitter” Call to Action on  your own WordPress or Ghost blog.

WordPress

I used the Bottom of Every Post Wordpress Plugin because I found that hacking the main wordpress loop caused my CTA to show up at the very bottom of a page rather than at the end of the post content. In addition, I made a small change to this plugin to make sure that it only appears on single posts rather than on the home page. Edit bottom-of-every-post/bottom_of_every_post.php from the WordPress Plugins Editor and update the method to be:

if( is_single() && file_exists( $fileName )){

 /* open the text file and read its contents */

 $theFile = fopen( $fileName, "r");
 $msg = fread( $theFile, filesize( $fileName ));
 fclose( $theFile );

 /* detect the old message in code to try and eradicate my name and #
 showing up on strange websites that are run by lazy people */

 if( $msg == "<p>Call for an estimate 724-498-1551<br><a href=\"mailto:corey.salzano@gmail.com\">corey.salzano@gmail.com</a></p>" ){
 $msg = "<p>Thank you for installing the Bottom of every post WordPress plugin. To find out how to change or remove this message, read <a href=\"http://wordpress.org/extend/plugins/bottom-of-every-post/installation/\">the instructions</a>.</p>";
 }

 /* append the text file contents to the end of `the_content` */
 return $content . stripslashes( $msg );
 } else{

 /* if `the_content` belongs to a page or our file is missing
 the result of this filter is no change to `the_content` */

 return $content;
 }

The key here is

is_single()

Next, edit bottom-of-every-post/bottom_of_every_post.txt and add your call to action. Mine looks like this.

<p>
 If you made it this far, you should probably follow me on twitter. :) 
 <a class="twitter-follow-button" href="https://twitter.com/levlaz"> Follow @levlaz</a>
</p>

<script>window.twttr = (function(d, s, id) {
 var js, fjs = d.getElementsByTagName(s)[0],
 t = window.twttr || {};
 if (d.getElementById(id)) return t;
 js = d.createElement(s);
 js.id = id;
 js.src = "https://platform.twitter.com/widgets.js";
 fjs.parentNode.insertBefore(js, fjs);

 t._e = [];
 t.ready = function(f) {
 t._e.push(f);
 };

 return t;
}(document, "script", "twitter-wjs"));</script>

You only need to replace your twitter username in the above example and it should work as is.

I used ghosts code-injection tool to get this to work for every page. I added the following two scripts to the Blog Footer.

<script> 
 article = document.getElementsByClassName('post-content');
 child = document.createElement('p');
 cta = '<p class="follow"> If you read this far, thank you! Follow me on Twitter to stay up to date on what the fuss is all about.<br />';
 link = '<a class="twitter-follow-button" href="https://twitter.com/tralevnet"> Follow @tralevnet</a></p>'
 child.innerHTML = cta + link;
 article[0].appendChild(child);
</script>

<script>
 window.twttr = (function(d, s, id) {
 var js, fjs = d.getElementsByTagName(s)[0],
 t = window.twttr || {};
 if (d.getElementById(id)) return t;
 js = d.createElement(s);
 js.id = id;
 js.src = "https://platform.twitter.com/widgets.js";
 fjs.parentNode.insertBefore(js, fjs);

 t._e = [];
 t.ready = function(f) {
 t._e.push(f);
 };

 return t;
}(document, "script", "twitter-wjs"));
</script>

In the example above you need to change your call to action as well as your twitter username for it to work. I also added a custom style to the Blog Header

<style> 
.follow {
 line-height: 1.5;
 width: 50%;
 padding: 15px;
 border: dashed 1px lightgrey;
 font-size: smaller;
 color: #555;
 font-family: sans-serif;
 font-weight: bold;
 }

 .twitter-follow-button {
 margin-top: 10px;
 }
</style>

That’s pretty much it. You can go wild with the style to your hearts content. I like this simple CTA and I am excited to see how effective it is on my own blogs.

Running Flask Tests without installing the app

The Flask Docs have a great section for testing. However they assume that you have the app installed with pip install -e . which I almost never do. (Maybe I should start?) I have had trouble with this approach. So one little hack to inject your app into the searchable python path is to do something like this:

export BLOG_PATH=$(pwd) && python tests/blog_tests.py

We use \$(pwd) so that this will “just work” no matter which computer you run it on (i.e. test, dev, other persons dev) Then in yourtests/blog_tests.py file you import your flask app like this:

sys.path.insert(0, os.environ.get('BLOG_PATH'))
from blog.blog import app, init_db

You can see the full details here: https://circleci.com/gh/levlaz/blog My biggest issue with installing the app in a traditional sense is that when I run it with apache mod_wsgi it does not seem to actually copy over the right folders and apache is not able to find templates or static files. So that sucks, and I am probably doing something wrong. But this approach works more or less no matter what so sometimes a hack is better than derping around with no solution (as I did for six hours yesterday).