levlaz

Posts tagged: python

Injecting Stuff into your Python Path

2017-04-13 12:00:18 ][ Tags: hacking python

Similar to a previous post where I wrote about how to run flask tests without installing your app, another common thing that you might want to be able to do is import your app from some arbitrary script. This is especially useful when running your app with apache mod_wsgi. This module expects the app to be installed globally or at least in the python path. Unless you install the app in a traditional sense this will not be true. The solution is just to inject the path prior to running your import statement like this.

sys.path.insert(0, '/var/www/blog')
from blog import app as application

This import will actually work.

Using the Flask CLI

2017-04-11 12:00:24 ][ Tags: hacking python flask

Who knew that flask had a cli? Previously I used to just use manage.py just like Django does it to "do stuff". The CLI is great, but again it follows the theme of kind of wanting you to install your flask app. (I really should do this). So in order to get your app to work you must point the FLASK_APP variable to the actual python file (not your app module). This is true even if you have a true python module. For instance.

export FLASK_APP = blog/blog.py
flask run

Works, while

export FLASK_APP = blog
flask run

Does not. Even though blog consists of:

blog/
  __init__.py
  blog.py

Running Flask Tests without installing the app

2017-04-08 12:00:06 ][ Tags: hacking testing python flask

The Flask Docs have a great section for testing. However they assume that you have the app installed with pip install -e . which I almost never do. (Maybe I should start?) I have had trouble with this approach. So one little hack to inject your app into the searchable python path is to do something like this:

export BLOG_PATH=$(pwd) && python tests/blog_tests.py

We use \$(pwd) so that this will "just work" no matter which computer you run it on (i.e. test, dev, other persons dev) Then in your tests/blog_tests.py file you import your flask app like this:

sys.path.insert(0, os.environ.get('BLOG_PATH'))
from blog.blog import app, init_db

You can see the full details here: https://circleci.com/gh/levlaz/blog My biggest issue with installing the app in a traditional sense is that when I run it with apache mod_wsgi it does not seem to actually copy over the right folders and apache is not able to find templates or static files. So that sucks, and I am probably doing something wrong. But this approach works more or less no matter what so sometimes a hack is better than derping around with no solution (as I did for six hours yesterday).

Standard Notes is a Better Project than Braindump

2017-01-17 23:18:14 ][ Tags: hacking python braindump

I released braindump to the world last year to much fanfare. After the initial excitement from being on HN died down, and the PR’s stopped rolling in, it became a personal project once again with very few users. Over the last few weeks I have made several attempts to fix the spaghetti mess that is the current code base by refactoring the current Flask implementation, then rewriting it completely in Django, and even started a branch to investigate rewriting the whole app in PHP using Laravel. Other commitments took precedence and Braindump remains in a fairly usable but not that special state.

Today on HN I read about a new project called Standard Notes which is the most exciting note related project that I have seen in a long time. It solves so many of the problems around cross platform compatibility that plague many other note tools. In addition its goals are to create a standard file format for simple, secure, and durable notes. Even more it has already created a platform, an ecosystem, that allows anyone to come and create additional applications, plugins, and use cases for notes.

These are some of the problems that I set out to tackle when I started braindump. After reading about Standard Notes, and using it for a few hours, I have decided that my time would be better spent contributing to that project instead of continuing to work on Braindump.

Working on Braindump has been amazing. I learned a ton, became a better programmer, and most of all had a lot of fun. I want to thank everyone who tried it, provided feedback, and sent patches. The source code for braindump will remain on GitHub but I would encourage you to try and contribute to the Standard Notes project along with me.

Outstanding Tutorial on writing GTK+ 3 GUI Applications with Python

2016-12-12 09:00:38 ][ Tags: hacking gtk python

GUI programming has always been a black art to me. The idea of event-driven applications with an endless infinite loop listening for events boggles my mind a bit and runs counter to my traditional understand of what algorithms look like.  This is something that I struggle the most with Javascript (which has the benefit of having most of the details abstracted away by the browser), and something that I have never been able to quite grasp using traditional GUI tools like GTK, Qt, and even .NET.

The trend these days is to just use Electron. It seems like more and more apps (Slack, Wordpress, Ghost, Postman, Visual Studio Code, etc.) are drinking the cross platform cool aid. I read this article which discusses some of the negative aspects of choosing Electron as a GUI framework. Drew makes some great points and the following paragraph inspired me to once again take a stab at actually learning a proper GUI framework.

Learn how to use GTK or Qt. Maybe Xwt is more up your alley. How about GNOME’s Vala thing? Learn another programming language. Learn Python or C/C++ or C#. Fun fact: it’ll make your JavaScript better, and once you have it in your toolbox you can make more educated decisions on the appropriate tool to use when you face your next problem.

Source: Electron considered harmful - Drew Devault's Blog

PyGObject (aka PyGI) is the new way of developing GTK+ 3 GUI applications in python. On Ubuntu, installing the python3-gi package is enough to get started making your application "do something". I found an excellent tutorial on using PyGI which does a great job explaining both basic and advanced concepts.

This tutorial gives an introduction to writing GTK+ 3 applications in Python.

Source: The Python GTK+ 3 Tutorial — Python GTK+ 3 Tutorial 3.4 documentation

In addition, this tutorial also has a section on how to use Glade, which is a GUI tool for building GUIs. For any substantial project hard coding the UI is going to get old pretty quickly.

The Python GObject Introspection API is massive. It solves a lot of common problems and also lets you work with a lot of existing GNOME applications. Looking forward to making something useful soon.

That UI Bug with Missing Data is a Security Issue

2016-11-16 20:47:06 ][ Tags: hacking python braindump security

This is (sometimes) a development blog, so I am going to write about some failed development of mine since writing about success is much less interesting. You know that UI bug that someone added to your GitHub Issues where there is some missing data? You know the one, it only happens in production, all of your tests pass, and you marked it as a low priority. Yeah, that one. It's probably a security bug and you should look into it right away. At least, that is the lesson I taught  myself yet again when I began to research this bug. Before

<a id="shared_note.id)" href=""{{">
{{current_user.notes.filter_by(id=shared_note.id).first().title}}
</a>

After

<a href="{{ url_for('main.note', id=shared_note.note_id) }}">
{{current_user.notes.filter_by(id=shared_note.note_id).first().title}}
</a>

The difference is very subtle, but the key issue here is shared_note.id vs shared_note.note_id; I released a feature a few weeks ago that showed you all of the notes that you have shared. Locally everything worked fine, but I noticed later on, once it was in production, that the note title was not showing up. This is, of course, due to the fact that rather than showing the title of the note with the ID shared_note.note_id (the foreign key linking to the note) I was showing the title for the note with the primary key of shared_note.id. The reason why this is a security issue is because this allows someone to share a bunch of notes and start seeing the titles for notes that they do not own. The reason why this worked locally is because I am only testing with a single user, with a single notebook, with a single note, and with a single shared note. This means that in this specific case all of the Primary Keys and Foreign Keys are usually "1" so everything just happens to work.

Key Takeaways

Show all Flask Routes

2016-08-28 21:02:21 ][ Tags: hacking python flask

One of my favorite things about rails is rake routes. This command will show you all of the routes that your application is currently aware of. I was searching for something similar in Flask and came across this snippet. Sadly, it did not work for me. I am using Python 3, so that may have something to do with it. In any case, I did not have time to dig into what was wrong, because just the other day I read this amazing introduction to functional python where my biggest take away was that as soon as you see multiple for loops, its time to use map. With my functional skills in tow, I wrote a simple one liner to print all of my routes.

@manager.command
def routes():
    import pprint
    pprint.pprint(list(map(lambda x: repr(x), app.url_map.iter_rules())))

Running this on braindump, shows me everything that I need to know.

[levlaz braindump]$ python3 manage.py routes
["<Rule '/auth/change-password' (GET, HEAD, OPTIONS, POST) -> "
'auth.change_password>',
"<Rule '/auth/change-email' (GET, HEAD, OPTIONS, POST) -> "
'auth.change_email_request>',
"<Rule '/auth/unconfirmed' (GET, HEAD, OPTIONS) -> auth.unconfirmed>",
"<Rule '/auth/register' (GET, HEAD, OPTIONS, POST) -> auth.register>",
"<Rule '/auth/confirm' (GET, HEAD, OPTIONS) -> auth.resend_confirmation>",
"<Rule '/auth/logout' (GET, HEAD, OPTIONS) -> auth.logout>",
"<Rule '/auth/login' (GET, HEAD, OPTIONS, POST) -> auth.login>",
"<Rule '/auth/reset' (GET, HEAD, OPTIONS, POST) -> "
'auth.password_reset_request>',
"<Rule '/empty-trash' (GET, HEAD, OPTIONS) -> main.empty_trash>",
"<Rule '/notebooks' (GET, HEAD, OPTIONS, POST) -> main.notebooks>",
"<Rule '/favorites' (GET, HEAD, OPTIONS) -> main.favorites>",
"<Rule '/settings' (GET, HEAD, OPTIONS) -> main.settings>",
"<Rule '/shutdown' (GET, HEAD, OPTIONS) -> main.server_shutdown>",
"<Rule '/archive' (GET, HEAD, OPTIONS) -> main.view_archive>",
"<Rule '/search' (GET, HEAD, OPTIONS) -> main.search>",
"<Rule '/trash' (GET, HEAD, OPTIONS) -> main.trash>",
"<Rule '/add' (GET, HEAD, OPTIONS, POST) -> main.add>",
"<Rule '/' (GET, HEAD, OPTIONS, POST) -> main.index>",
"<Rule '/static/bootstrap/<filename>' (GET, HEAD, OPTIONS) -> "
'bootstrap.static>',
"<Rule '/auth/change-email/<token>' (GET, HEAD, OPTIONS) -> "
'auth.change_email>',
"<Rule '/auth/confirm/<token>' (GET, HEAD, OPTIONS) -> auth.confirm>",
"<Rule '/auth/reset/<token>' (GET, HEAD, OPTIONS, POST) -> "
'auth.password_reset>',
"<Rule '/delete-forever/<id>' (GET, HEAD, OPTIONS, POST) -> "
'main.delete_forever>',
"<Rule '/notebook/<id>' (GET, HEAD, OPTIONS) -> main.notebook>",
"<Rule '/notebook/<id>' (DELETE, OPTIONS) -> main.delete_notebook>",
"<Rule '/favorite/<id>' (GET, HEAD, OPTIONS, POST) -> main.favorite>",
"<Rule '/restore/<id>' (GET, HEAD, OPTIONS, POST) -> main.restore>",
"<Rule '/archive/<id>' (GET, HEAD, OPTIONS) -> main.archive>",
"<Rule '/delete/<id>' (GET, HEAD, OPTIONS, POST) -> main.delete>",
"<Rule '/static/<filename>' (GET, HEAD, OPTIONS) -> static>",
"<Rule '/share/<id>' (GET, HEAD, OPTIONS, POST) -> main.share>",
"<Rule '/note/<id>' (GET, HEAD, OPTIONS) -> main.note>",
"<Rule '/edit/<id>' (PUT, OPTIONS) -> main.edit>",
"<Rule '/tag/<name>' (GET, HEAD, OPTIONS) -> main.tag>"]

Now, all that is left is for me to figure out why my API routes are missing. :)

Recreating Foreign Keys with Alembic

2016-05-30 18:59:18 ][ Tags: hacking python databases

Alembic is a great tool for keeping track of schema changes in python applications. I am using it to manage DB migrations for braindump along with Flask SQL Alchemy as my ORM. One challenge is managing proper foreign key constraints. By default if you define a foreign key relationship in your schema definition it will not generate the proper migration code. For example, in braindump we have a one to many relationship between users and notes.

class User(UserMixin, db.Model):
    __tablename__ = 'users'
    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String(254), unique=True, index=True)
    password_hash = db.Column(db.String(256))
    confirmed = db.Column(db.Boolean, default=False)
    avatar_hash = db.Column(db.String(32))
    created_date = db.Column(db.DateTime(), default=datetime.utcnow)
    updated_date = db.Column(db.DateTime(), default=datetime.utcnow)

    notes = db.relationship(
        'Note', backref='author',
        lazy='dynamic', cascade="all, delete-orphan")

Even though we define the cascade behavior using SQLAlchemy. When we generate the migration with alembic we get something like this:

sa.ForeignKeyConstraint(['author_id'], ['users.id'], ),

Notice how we are missing the ondelete action. What we actually want is something like this:

sa.ForeignKeyConstraint(['author_id'], ['users.id'], ondelete='CASCADE')

Running the default migration will not create the proper relationship in your database and in our case we are not able to delete a user until we have deleted all of the related records as well. There are two ways to fix this. If you catch this before running your migration adding ondelete='CASCADE' will create the proper relationship. If you are like me, and do not catch this, then you will need to run a second migration to remove and recreate these keys. The migration code to do this is shown below:

from alembic import op
import sqlalchemy as sa

def upgrade():
    with op.batch_alter_table("notes") as batch_op:
        batch_op.drop_constraint(
            "notes_author_id_fkey", type_="foreignkey")
    op.create_foreign_key(
        "notes_author_id_fkey", "notes", "users",
        ["author_id"], ["id"], ondelete="CASCADE")

Now you have the proper foreign key constraints and the CASCADE action exists in the DB.

pip-save NPM Style Package Management for Python

2016-04-21 18:51:39 ][ Tags: python

One of my favorite things about npm is how simple it makes it to manage production, testing, and development dependencies with the npm install --save command. I was browsing through the latest packages on PyPI and I came across pip-save which does basically the same thing as npm --save as a wrapper around pip.

From the Project README:

Currently its a big pain while installing new dependencies using pip. After installing the dependency, you need to figure out the version number and then manually add it to your requirements file. pip-save allows you to install/uninstall any dependecy and automatically add/remove it to/from your requirements file using one command only.

I cannot count the number of times where I have read a doc for some package and it said that installation was as simple as pip install foo. Sure, installing the package is that easy, but keeping track of versions is a whole different story.

One approach is to dump every single package into your requirements.txt with pip freeze > requirements.txt but this is troublesome because it also includes dependencies which clutter up the requirements.txt file.

What I then end up doing is finding the package on either GitHub or PyPI, figuring out what the latest version is, putting the package with the version into my requirements.txt file and then running pip install -r requirements.txt.

pip-save does exactly what it says on the tin and solves this annoying workflow in one step. It installs whatever package you want, and adds it to your requirements.txt with the version that you installed. Simple, Rustic, Yes.

My First Pip Package

2016-04-21 18:50:24 ][ Tags: hacking python

I finally got around to packaging up pg2cf and I am excited to see it out in the wild on pypi. Python packaging is pretty straightforward and this makes distribution of this tool much easier for us internally.

I could not have done it without this awesome python packaging guide.

Using setuptools also makes a lot of other things easier. For instance running tests is as simple as python setup.py test and installing the package locally can be done with python setup.py install. The main benifit of course is that now pg2cf is an executable so you once it is installed you can just run it.

The only "gotcha" that I ran into was that setuptools does not support markdown. Which makes it kind of weird since Markdown is now the standard for README and other documentation on GitHub.

I worked around this by using pandoc, this way I am able to convert my README.md to README.rst easily with pandoc README.md -o README.rst and then use that for PyPI.

Introducing ezBadge

2015-07-10 17:55:51 ][ Tags: hacking python

A few months ago I wrote about my struggles with Markdown Image Links. The entire reason why I was working on that before was to put cute little badges in my Github repos. Today I took this a whole step forward and wrote a small web application that just “does the needful” for you. ezBadge makes it easy to transform a regular Github Repo URL into a beautiful badge by writing the markdown for you and allowing you to just copy and paste it into your README.md file or anywhere else where Markdown is supported. You just feed it a valid Github repo URL and it spits out the Markdown along with a preview of what you will see if you put that markdown somewhere. So far, the only type of Badge that it makes is a CircleCI badge but I plan on adding more in the near future and if there is a badge that you would like to see, please open up an issue for this project.

Module Import Errors in Python when Executing a Program

2015-06-24 17:55:12 ][ Tags: hacking python

I ran into a weird issue today with a python program where even though I have the pyyaml module installed python would complain that ImportError: No module named yaml when I tried to execute my program. I have ran into this in the past and the solutions posted online vary greatly and usually didn’t work for me. I think I have found the root cause of this and wanted to quickly share it here. I am running Python 2.7.9 on OS X Mavericks. Pyyaml was installed via pip and works just fine. The interesting part about this is that when I execute the program by running python program.py everything works fine. If I chmod +x program.py to make it executable and then try to execute it by running ./program.py then this issue begins to arise. The key issue happens to be the header that tells the OS where to find python. When you run python from a shell in OS X it is looking to /usr/local/bin/python to find and execute python. You can verify that this is true by running which python in a terminal. When you execute a python program without explicitly calling python then it will look for python in your file header. The first line of a file typically looks like this #!/usr/bin/python. This was the cause for the descrepancy in my case. Since the python that is called when I say python is different than the python that I defined in my header the module was not able to load. Pip installs all modules wherever your main python is. The fix for this issue is to change the header from #!/usr/bin/python to #!/usr/bin/env python This way it will look to whatever the default python installation is, and in the event that you are running a virtual environment there will not be any clashing. This seems like one of those tricky bash gotcha’s that happen sometimes. Using #!/usr/bin/env python by default is a safe bet to make sure you don’t run into these types of issues.

Programming Sockets in Python

2015-01-25 02:35:54 ][ Tags: hacking python

EDIT: I am sorry for derping out of control. When I initially published this post it was called “Programming Web Sockets in Python”, this is just flat out wrong. What we are making here is just a regular socket. Web Sockets and regular sockets are similar but are certainly not the same thing. I hope you will still find this useful! Sockets are pretty much the basis of how applications work on the Internet. Python makes it super easy to get started programming sockets. In this brief introduction we will create a simple server that greets the user when it receives incoming requests from the client application. Due to my recent obsession with Linux Containers we will also be implementing this inside of two containers. Containers make it really simple to simulate a network because you can create additional hosts in seconds.

Creating your Containers

I am running Ubuntu 14.04. So creating two additional containers can be achieved by running the following as the root user.

lxc-create -t download -n pyServer # Choose ubuntu, trusty, amd64 when prompted # Then clone the first container lxc-clone -o pyServer -n pyClient

Running the Server

Now that we have created our containers lets jump into our server container and fire up our simple server application. We can start up the container by issuing the following command as root: lxc-start -n pyServer -d, this will start the container as a daemon. Let’s go ahead and get into by attaching the container. I like to do this inside of screen so that we can easily get in and out of the container. Create a screen session screen -dRR pyServer and once inside the screen attach the container lxc-attach -n pyServer Once we are inside the container we need to install python and launch our simple server.

apt-get install python vim pyServer.py

Inside of vim (or your favorite text editor) we need to enter the following simple python code.

from socket import *  serverPort = 12000  serverSocket = socket(AF_INET, SOCK_DGRAM)  serverSocket.bind(('', serverPort))  print "The server is ready to rock and roll!"  while 1:     name, clientAddress = serverSocket.recvfrom(2048)     response = "Hello " + str(name) + "! You are really good at socket programming"     serverSocket.sendto(response, clientAddress)

The code should be pretty straightforward. We are creating a new serverSocket that is bound to port 12000. When it receives requests (which include a name) it responds with an encouraging message. Fire up this server by running python pyServer.py if all goes well you should see a message that states This server is ready to rock and roll! Exit the container (and the screen session) by pressing Ctrl+a and Ctrl+d

Running the Client

Now that we have our server up and running, lets get our client working as well. Before we move forward, lets grab the IP address of our server container because we will need it soon. You can get the IP by running lxc-ls --fancy. Launch the client container, attach it in screen, and install python in the same way that we did previously.

lxc-start -n pyClient -d screen -dRR pyClient lxc-attach -n pyClient apt-get install python vim pyClient.py

In vim, lets create the pyClient.py program by entering the following code.

from socket import *  # Replace the IP address in serverName with the IP of your container that you      grabbed previously. serverName = '10.0.3.211' serverPort = 12000 clientSocket = socket(AF_INET, SOCK_DGRAM)  name = raw_input('Please enter your name:')  clientSocket.sendto(name, (serverName, serverPort)) response, serverAddress = clientSocket.recvfrom(2048) print response clientSocket.close()

This code is also pretty straightforward. It asks the user for their name, sends it to the server, and prints the response. You can try this out now! Save the file and execute your python program by running python pyClient.py. After entering your name and pressing enter you should see a response from your server with the encouraging message. This was a pretty trivial exercise, but we can quickly see that we can expand upon this basic code to create much more interesting and complex applications. We can also leverage the power and simplicity of LXC to create a simulated large network for distributed applications.