Slow Python Script and Using Pipenv with AWS Lambda
I’m working on improving a python script I wrote to get a list of old posts from a wordpress website. Basically I want to be able to see what post I wrote X years ago on this day for any wordpress site.
This script uses the wonderful requests library and the very powerful public Wordpress API.
I am also using pipenv for the first time and its wonderful. I wish I started using this tool years ago.
What it Does Right Now
- Takes a dictionary of sites and iterates over each one
- Prints out to the console
print("1 year ago I wrote about {0} {1}".format(p['title']['rendered'], p['link']))
if years_ago > 1:
print("{0} years ago I wrote about {1} {2}".format(years_ago, p['title']['rendered'], p['link']))
The Script is Super Slow
You can time how long a script takes on OS X using thetime
command.
Levs-iMac:OldPosts levlaz$ time python old_posts.py
1 year ago I wrote about Thoughts on “Sacramento Renaissance” https://tralev.net/thoughts-on-sacramento-renaissance/
real 0m11.192s
user 0m0.589s
sys 0m0.060s
Plans for Optimization
- Use Redis (or something) to cache the results.
- Get rid of some of the for loops if we can.
Plans for Usage
- Deploy to AWS (Labmda?)
- Have this run on a Cron Job every day (using CloudWatch)
Plans for Additional Features
I want to share all of the posts from that day on social media. Instead of plugging in all of the various accounts that I need I am planning on using the Buffer API to post everywhere at once and queue up posts so that it does not fire off a bunch of stuff at the same time in the event that there are many posts for that day.This will involve doing some sort of Outh dance because I don’t think that Buffer offers using personal access tokens.
I'll Just Use Lambda
Famous last words.It’s not the worst thing in the world, but when you are using the amazing pipenv tool you have to go track down where the site-packages are located and zip them up in order to ship your code to AWS Lambda.
Unsurprisingly someone opened a feature request for this, but the solution in the comments works just fine.
I wrote a little bash script that is being called through a Makefile to zip up the site-packages along with the core python code in preparation to ship it off to AWS Lambda.
Bash Script to Zip Up Site-Packages
Makefile
.PHONY: package
package:
sh package.sh
Thank you for reading! Share your thoughts with me on bluesky, mastodon, or via email.
Check out some more stuff to read down below.
Most popular posts this month
- 2024
- Reinstalling Windows at 1am
- SQLite DB Migrations with PRAGMA user_version
- My Custom Miniflux CSS Theme
- How to Disable Wayland in Debian Testing
Recent Favorite Blog Posts
This is a collection of the last 8 posts that I bookmarked.
- Underused Techniques for Effective Emails from Refactoring English
- Death by a thousand slops from daniel.haxx.se
- The AGI economy is coming faster than you think from Freethink
- Rolling the ladder up behind us from Xe Iaso's blog
- In Praise of “Normal” Engineers from charity.wtf
- Reports of Bluesky's death have been greatly exaggerated from The Torment Nexus
- What Would a Kubernetes 2.0 Look Like from matduggan.com
- We Can Just Measure Things from Armin Ronacher's Thoughts and Writings
Articles from blogs I follow around the net
Occupation and Preoccupation
Here’s Jony Ive in his Stripe interview: What we make stands testament to who we are. What we make describes our values. It describes our preoccupations. It describes beautiful succinctly our preoccupation. I’d never really noticed the connection between th…
via Jim Nielsen’s Blog July 17, 2025What do we do when the facts don't matter?
As a journalist by training, I have what you might call an addiction to facts. I believe that there are things that are true and things that are not, and that we can determine one from the other through observation, rational thought, and meticulous resear…
via The Torment Nexus July 17, 2025Back to the Future
This past weekend, I decided to take a trip down memory lane and take a look at the evolution of this blog’s design over the years (with a little help from the Wayback Machine). While I’ve really enjoyed the challenge of making an obsessively backwards co…
via flower.codes July 17, 2025Generated by openring