Slow Python Script and Using Pipenv with AWS Lambda

I’m working on improving a python script I wrote to get a list of old posts from a wordpress website. Basically I want to be able to see what post I wrote X years ago on this day for any wordpress site.

This script uses the wonderful requests library and the very powerful public Wordpress API.

I am also using pipenv for the first time and its wonderful. I wish I started using this tool years ago.

What it Does Right Now

1. Takes a dictionary of sites and iterates over each one
2. Prints out to the console

print("1 year ago I wrote about {0} {1}".format(p['title']['rendered'], p['link']))
if years_ago > 1:
print("{0} years ago I wrote about {1} {2}".format(years_ago, p['title']['rendered'], p['link']))

The Script is Super Slow

Levs-iMac:OldPosts levlaz$ time python old_posts.py
1 year ago I wrote about Thoughts on “Sacramento Renaissance” https://tralev.net/thoughts-on-sacramento-renaissance/
real	0m11.192s
user	0m0.589s
sys	0m0.060s

Plans for Optimization

1. Use Redis (or something) to cache the results.
2. Get rid of some of the for loops if we can.

Plans for Usage

1. Deploy to AWS (Labmda?)
2. Have this run on a Cron Job every day (using CloudWatch)

Plans for Additional Features

This will involve doing some sort of Outh dance because I don’t think that Buffer offers using personal access tokens.

I'll Just Use Lambda

It’s not the worst thing in the world, but when you are using the amazing pipenv tool you have to go track down where the site-packages are located and zip them up in order to ship your code to AWS Lambda.

Unsurprisingly someone opened a feature request for this, but the solution in the comments works just fine.

I wrote a little bash script that is being called through a Makefile to zip up the site-packages along with the core python code in preparation to ship it off to AWS Lambda.

Bash Script to Zip Up Site-Packages

SITE_PACKAGES=$(pipenv --venv)/lib/python3.6/site-packages
DIR=$(pwd)

  Make sure pipenv is good to go
  
    
  

pipenv install
cd $SITE_PACKAGES
zip -r9 $DIR/OldPosts.zip *
cd $DIR
zip -g OldPosts.zip old_posts.py

Makefile

.PHONY: package
package:
sh package.sh