programming

Slow Python Script and Using Pipenv with AWS Lambda

I’m working on improving a python script I wrote to get a list of old posts from a wordpress website. Basically I want to be able to see what post I wrote X years ago on this day for any wordpress site.

This script uses the wonderful requests library and the very powerful public WordPress API.

I am also using pipenv for the first time and its wonderful. I wish I started using this tool years ago.

What it Does Right Now

  1. Takes a dictionary of sites and iterates over each one
  2. Prints out to the console
print("1 year ago I wrote about {0} {1}".format(p['title']['rendered'], p['link']))
if years_ago > 1:
print("{0} years ago I wrote about {1} {2}".format(years_ago, p['title']['rendered'], p['link']))

The Script is Super Slow

You can time how long a script takes on OS X using the time command.

Levs-iMac:OldPosts levlaz$ time python old_posts.py
1 year ago I wrote about Thoughts on “Sacramento Renaissance” https://tralev.net/thoughts-on-sacramento-renaissance/

real	0m11.192s
user	0m0.589s
sys	0m0.060s

I know why its slow. Because I have like 6 for loops and a bunch of other inneficiencies. In addition, the requests are not cached anywhere so it has to get the entire JSON load each time that the script runs.

Plans for Optimization

  1. Use Redis (or something) to cache the results.
  2. Get rid of some of the for loops if we can.

Plans for Usage

  1. Deploy to AWS (Labmda?)
  2. Have this run on a Cron Job every day (using CloudWatch)

Plans for Additional Features

I want to share all of the posts from that day on social media. Instead of plugging in all of the various accounts that I need I am planning on using the Buffer API to post everywhere at once and queue up posts so that it does not fire off a bunch of stuff at the same time in the event that there are many posts for that day.

This will involve doing some sort of Outh dance because I don’t think that Buffer offers using personal access tokens.

I’ll Just Use Lambda

Famous last words.

It’s not the worst thing in the world, but when you are using the amazing pipenv tool you have to go track down where the site-packages are located and zip them up in order to ship your code to AWS Lambda.

Unsurprisingly someone opened a feature request for this, but the solution in the comments works just fine.

I wrote a little bash script that is being called through a Makefile to zip up the site-packages along with the core python code in preparation to ship it off to AWS Lambda.

Bash Script to Zip Up Site-Packages

SITE_PACKAGES=$(pipenv --venv)/lib/python3.6/site-packages
DIR=$(pwd)

# Make sure pipenv is good to go
pipenv install

cd $SITE_PACKAGES
zip -r9 $DIR/OldPosts.zip *

cd $DIR
zip -g OldPosts.zip old_posts.py

Makefile

.PHONY: package

package:
	sh package.sh

This should just work™.

Standard
programming

What is GlassFish?

I jumped down another rabbit hole trying to figure out how to get started with java ee without using an ide. Although IDE’s are very handy when it comes to Java development, they also are sometimes a crutch. For instance, if you want to transition to CI, do you actually know what commands the IDE runs when you right click and run tests?

First, I have no idea what Java EE actually is. There is something called GlassFish, which is an open source Java EE “reference implementation”. It also the same thing that is installed when you go to the main Java EE website.

Java EE does not support the latest Java JDK 1.9. On my Mac I had a tough time trying to get two versions of Java to run at the same time.

I think 99.9% of all tutorials about getting started with Java EE include using Netbeans or Eclipse. I wanted to write one that used the CLI. This involves using maven.

Maven has a concept called “archetypes” which creates the necessary directory structure for a new Java project. The main problem is that I could not find a bare bones archetype definition.

At the end of the day, I dug deep into the rabbit hole and came up empty. I will figure this out at some point and write a blog post about it.

Standard
tech

Learn Kubernetes with Interactive Tutorials

I wanted to get a deeper understanding of how Kubernetes actually works, so I started to work through the tutorials on the kubernetes documentation website.  Kubernetes is a container orchestration system that creates some standard tooling for deploying, scaling, and managing containers at scale.

The tutorials themselves, are amazing.

The tutorials use Katacoda to run a virtual terminal in your web browser that runs Minikube, a small-scale local deployment of Kubernetes that can run anywhere.

At a high level kubernetes allows you to deploy a cluster of resources as a single unit without having to really think about the underlying individual hosts. It follows a master -> node model where there is a centralized control point for managing your cluster and worker nodes that perform the actions that your application needs.

Kubernetes supports running both Docker containers and rkt containers. I’m pretty familiar with Docker. I learned more than I ever wanted to over the last few years of working at CircleCI. I have never used rkt, but am looking forward to learning more in the future.

It is really neat that you can simulate a production-like instance on your local computer using minikube. This is a great way to learn kubernetes as well as be able to do local development.

Kubernetes docs has some interactive tutorials that allow you to get your hands dirty with Kubernetes without having to install anything. These tutorials are powered by KataCoda, a tool that I am not familiar with. This is a neat web service that allows you to learn new technologies in your browser.

Kubernetes in your Browser

Kubernetes in your Browser

The first tutorial teaches you how to use minikube, and the kubectl cli to create a new cluster.

One of the most amazing parts of kubernetes to me is the self-healing aspect. For example once you have defined what your application stack consists of, if a node happens to go down then kubernetes will automatically replace it with another instance.

Not only does the interactive online tutorial allow you to use a real kubernetes cluster from within your browser, you can even preview the web UI portion of the cluster as well as viewing your application running.

Kubernetes Web UI

Kubernetes Web UI

This is such a great way to learn.

Standard