Docker Based Development Environment for Packaging

I started a new repo called sid-builder which provides a docker based development environment for debian packaging. This project follows all of the best practices that were outlined in the Guide for New Maintainers. The benefit of this approach is that you should be able to easily reproduce your development environment on any machine that supports docker. It also allows you to easily run sid without worrying about breaking your system.

I have not actually built any packages using this setup just yet, but I am going to give it a try over the next few days and report back here.

Becoming a Debian Developer

I’ve been using debian for as long as I remember. I’ve always wanted to play a more active role in debian development but for whatever reason I never got around to it. Now that I am 30, older, wiser, I am starting a new push to become a debian developer.

So far I have gotten involved in the debian-qa team. Specifically I have been working on fixing some of the newcomer bugs on the distro-tracker project. It’s actually been really fun. The code base is django which I am pretty comfortable with. For the first time in many years, I have been rushing home after work so that I could keep hacking on the bugs that I am working on.

Code contributions are one thing. The thing that I am going to need more practice on is debian packaging itself. This is pretty complex process and I think that becoming someone who can debug and perform packaging issues would bring a lot of value to the project.

I am going to start working on some of the RC bugs (which typically involve at least re-packaging software) to get more comfortable with how other folks have been doing packaging.

In the next few months I would love to bring a new package through it’s full lifecycle.

I hope to be able to look back on this blog post in the future and see how far I have come.

Dealing With Flakey CI Commands With a Retry Loop in Bash

One of the most frustrating things to deal with in Continuous Integration is flakey commands. Whether it’s flakey tests, or intermittent networking issues, when your build fails for issues outside of your control not only does it cause frustration, it reduces the trust in your CI process.

One strategy for dealing with this type of issue is to introduce some retry logic into your commands. This can easily be accomplished with good old bash.

For example, pretend that I have $FLAKEY_COMMAND and I want to retry it three times before finally failing my build. I could wrap the whole thing up in a bash loop like this.

counter=1
max=3
$FLAKEY_COMMAND

while [[ $? -ne 0 && $counter -lt $max ]]; do
    counter=$((counter+1))
    $FLAKEY_COMMAND
done

This script will run my command, if the exit code (the output of $?) is non zero (i.e something went wrong) and my counter is less than three, then it will retry the command. You can increase or decrease the number of attempts by adjusting the max variable.

This is not a foolproof strategy, but is one approach to handle flakey commands in your CI pipeline.

Notes On Installing CentOS 7 Server

I’ve been on a fedora kick lately, and naturally for my server needs I am using CentOS 7. I have never really used CentOS in production and there are a couple of gotchas that I ran into while getting everything set up that I wanted to jot down for future reference and also in case its useful to someone else.

Install Some Useful Packages

The default install of CentOS is pretty bare bones. I installed several packages to make it a bit more usable.

sudo yum install wget unzip git htop vim epel-release

I tried to get away without installing epel[3], but it’s too darn useful.

Enable SSH Login

I have no idea how or why this works (I think it might have something to do with SELinux), but in order to be able to SSH into your server you need to set up your authorized keys file like this[1]:

mkdir .ssh
chmod 755 .ssh/

# copy your id_rsa.pub file to .ssh/authorized_keys 
# you can do this with a text editor, or if its on github
# download it with wget https://github.com/$USER.keys and 
# rename it to be the .ssh/authorized_keys file 

chmod 600 .ssh/authorized_keys
sudo restorecon -R -v .ssh

After that be sure to set PasswordAuthentication no in /etc/ssh/sshd_config and restart the sshd service for this to take effect. sudo systemctl restart sshd.service.

Install Docker

I used a convenience script from the main docker docs [2], I also added myself to the docker user group in order be able to run docker commands without root.

curl -fsSL get.docker.com -o get-docker.sh
sudo sh get-docker.sh
sudo usermod -aG docker $USER

I am not using the root user to log in, I am using my own user.

If you want to use docker-compose, then (assuming you installed epel-release) you should install pip and docker-compose.

sudo yum install python34-pip
sudo pip3 install docker-compose

Allow outside connections

CentOS uses firewalld[4], it is a bit more complex than what I am used to with UFW, but certainly easier to use than iptables.

You can allow traffic on http and https with the following commands.

sudo firewall-cmd --zone=public --add-service=http
sudo firewall-cmd --zone=public --add-service=https

References
[1]Creating .ssh folder
[2]Install Docker on CentOS
[3]Extra Packages for Enterprise Linux
[4]DO firewalld guide

Installing Sage Math on Fedora 28

Sage Math is a massive collection of open source mathematical tools. I am using it as a part of going through the Free Linear Algebra Book.

On a fresh install of Fedora 28, when you run dnf install sagemath you will install dozens of different packages. However, when you try to launch the sage math program you will get an error message saying that sage math crashed.

┌────────────────────────────────────────────────────────────────────┐
│ SageMath version 8.0, Release Date: 2017-07-21                     │
│ Type "notebook()" for the browser-based notebook interface.        │
│ Type "help()" for help.                                            │
└────────────────────────────────────────────────────────────────────┘

**********************************************************************

Oops, Sage crashed. We do our best to make it stable, but...

A crash report was automatically generated with the following information:
  - A verbatim copy of the crash traceback.
  - A copy of your input history during this session.
  - Data on your current Sage configuration.

It was left in the file named:
    '/home/levlaz/.ipython/Sage_crash_report.txt'
If you can email this file to the developers, the information in it will help
them in understanding and correcting the problem.

You can mail it to: sage-support at sage-support@googlegroups.com
with the subject 'Sage Crash Report'.

If you want to do it now, the following command will work (under Unix):
mail -s 'Sage Crash Report' sage-support@googlegroups.com < /home/levlaz/.ipython/Sage_crash_report.txt

In your email, please also include information about:
- The operating system under which the crash happened: Linux, macOS, Windows,
  other, and which exact version (for example: Ubuntu 16.04.3, macOS 10.13.2,
  Windows 10 Pro), and whether it is 32-bit or 64-bit;
- How Sage was installed: using pip or conda, from GitHub, as part of
  a Docker container, or other, providing more detail if possible;
- How to reproduce the crash: what exact sequence of instructions can one
  input to get the same crash? Ideally, find a minimal yet complete sequence
  of instructions that yields the crash.

To ensure accurate tracking of this issue, please file a report about it at:
http://trac.sagemath.org

Hit <Enter> to quit (your terminal may close):

If you inspect the error file found in $HOME/.ipython/Sage_crash_report.txt you will see that there is a missing python dependency.

ImportError: No module named cypari2.gen

You can fix this error by installing the missing dependency with:

sudo dnf install python2-cypari2

Now you should be able to launch Sage Math without it crashing.

Read All of Hacker News With the hanopener Extension

I’ve been reading Hacker News obsessively lately. In the past I would skip the top posts every couple of days, lately I have been reading every new and top article. In order to achieve this I would go to the main website, and click on every single link on the front pages.

After doing this for a few days I realized that I should probably write some Javascript to automate this entire process. So this afternoon I whipped up the hanopener chrome extension.

Initially I made it a python CLI script, but then realized that this would probably make more sense as a chrome extension.

Be warned: This extension is super obnoxious and will open up 60 chrome tabs every time you click on the icon, which is not always a good time depending on your computer.

It’s available on the chrome webstore and as a Firefox Add-On now.

UPDATE @ 5/28
Added link to Firefox Add On

Deploying an Angular 6 Application to Netlify

Netlify is an excellent platform for building, deploying, and managing web applications. It supports automated deployment using GitHub webhooks and also provides some advanced features such as custom domains and HTTPS all for free. Deploying a Static Site to Netlify is a breeze. Although it does support running Angular JS applications, there are a couple gotchas in the deployment process that I had to wrangle together from various blog posts in order to get things to work.

Enable Redirects

The first issue that I ran into was after I deployed my site to Netlify, whenever I would click on an Angular link, I would get a 404 page.

Netlify Page Not Found
Looks like you’ve followed a broken link or entered a URL that doesn’t exist on this site.

Getting this to work is pretty simple. Ultimately you just need a file called _redirects in the root of your web project. In order to get angular to create this you need to do the following things. This file will send all URL’s to the root of your application which allows the Angular router to kick in and do its thing.

  1. Create a _redirects file in the src directory of your angular project.

    For most basic sites it should look something like this.

    # src/_redirects
    
    /*  /index.html 200
    
  2. Add this file to your angular.json file.

    Your angular.json file serves as a configuration for many different aspects of the angular CLI. In order to get this file into the root of your output directory you must define the file here. A snippet of my file is shown below. Update this configuration file and push all of your changes back up to GitHub.

    {
    "$schema": "./node_modules/@angular/cli/lib/config/schema.json",
    "version": 1,
    "newProjectRoot": "projects",
    "projects": {
        "flagviz": {
        "root": "",
        "sourceRoot": "src",
        "projectType": "application",
        "prefix": "app",
        "schematics": {},
        "architect": {
            "build": {
            "builder": "@angular-devkit/build-angular:browser",
            "options": {
                "outputPath": "dist/flagviz",
                "index": "src/index.html",
                "main": "src/main.ts",
                "polyfills": "src/polyfills.ts",
                "tsConfig": "src/tsconfig.app.json",
                "assets": [
                "src/favicon.ico",
                "src/assets",
                "src/_redirects"
    
    ... rest of file 
    

Configure your Netlify Project

Now that you have the redirects file in place. You can set up your project for automatic deployment with GitHub and Netlify.

Once you have logged into Netlify, click on New Site From Git and find the name of your project.

New Site from GitHub

Configure Build Settings

The last step is to configure your build settings.

For Build command you should enter ng build --prod.

For Publish directory you should enter dist/$NAME_OF_YOUR_PROJECT.

Netlify Build Settings

Be sure to replace $NAME_OF_YOUR_PROJECT with the actual name of your project.

Now you can click on Deploy site and once the initial deployment has completed you should see your new angular application running on Netflify with a working routing system.

Install Terraform on an Ubuntu Server

Terraform by Hashicorp is a powerful tool that you can use to manage your infrastructure as code. It is distributed as a single binary so getting it installed on Ubuntu is a breeze.

  1. Assuming you are on a “standard” computer or server. From the downloads page copy the URL for the 64-bit Linux package. At the time of writing this was: https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
  2. SSH into your ubuntu server and execute wget https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
  3. Unzip this file (you may need to install the unzip package with sudo apt-get install unzip)
  4. Move the file to the /usr/local/bin directory with sudo mv terraform /usr/local/bin/

You can confirm that this works by typing in terraform -version in your terminal. Your output should look something like this.

ubuntu@ip-172-26-5-139:~$ terraform -version
Terraform v0.11.7

You should now be able to execute the terraform command from anywhere and manage your infrastructure as code.

Using Microsoft Power BI With PostgreSQL

Microsoft Power BI is an advanced business intelligence suite that allows you to perform robust data analysis from a variety of different data sources. One common data source is PostgreSQL. Although Microsoft PowerBI does support PostgreSQL, getting started can be a bit tricky because there is no great documentation.

If you try to connect to PostgreSQL with a fresh installation of PowerBI you will receive the following error message.

https://www.postgresql.org/

This connector requires one or more additional components to be installed before it can be used.

If you click on the Learn more link, it will take you to the GitHub repository for the Npgsql library, which is a windows driver for Postgres.

If you download the latest .msi file and run through the default installation, you will continue to receive the same error message in Power BI. In order to get this to work you must select the Npgsql GAC Installation option as shown in the screenshot below.

Npgsql GAC Installation Option

Once you have installed the Npgsql GAC Installation, you can restart Microsoft Power BI and you should now be able to connect to a PostgreSQL database as a data source.

PostgreSQL connection window in Microsoft Power BI
PostgreSQL connection window in Microsoft Power BI

Slow Python Script and Using Pipenv with AWS Lambda

I’m working on improving a python script I wrote to get a list of old posts from a wordpress website. Basically I want to be able to see what post I wrote X years ago on this day for any wordpress site.

This script uses the wonderful requests library and the very powerful public WordPress API.

I am also using pipenv for the first time and its wonderful. I wish I started using this tool years ago.

What it Does Right Now

  1. Takes a dictionary of sites and iterates over each one
  2. Prints out to the console
print("1 year ago I wrote about {0} {1}".format(p['title']['rendered'], p['link']))
if years_ago > 1:
print("{0} years ago I wrote about {1} {2}".format(years_ago, p['title']['rendered'], p['link']))

The Script is Super Slow

You can time how long a script takes on OS X using the time command.

Levs-iMac:OldPosts levlaz$ time python old_posts.py
1 year ago I wrote about Thoughts on &#8220;Sacramento Renaissance&#8221; https://tralev.net/thoughts-on-sacramento-renaissance/

real	0m11.192s
user	0m0.589s
sys	0m0.060s

I know why its slow. Because I have like 6 for loops and a bunch of other inneficiencies. In addition, the requests are not cached anywhere so it has to get the entire JSON load each time that the script runs.

Plans for Optimization

  1. Use Redis (or something) to cache the results.
  2. Get rid of some of the for loops if we can.

Plans for Usage

  1. Deploy to AWS (Labmda?)
  2. Have this run on a Cron Job every day (using CloudWatch)

Plans for Additional Features

I want to share all of the posts from that day on social media. Instead of plugging in all of the various accounts that I need I am planning on using the Buffer API to post everywhere at once and queue up posts so that it does not fire off a bunch of stuff at the same time in the event that there are many posts for that day.

This will involve doing some sort of Outh dance because I don’t think that Buffer offers using personal access tokens.

I’ll Just Use Lambda

Famous last words.

It’s not the worst thing in the world, but when you are using the amazing pipenv tool you have to go track down where the site-packages are located and zip them up in order to ship your code to AWS Lambda.

Unsurprisingly someone opened a feature request for this, but the solution in the comments works just fine.

I wrote a little bash script that is being called through a Makefile to zip up the site-packages along with the core python code in preparation to ship it off to AWS Lambda.

Bash Script to Zip Up Site-Packages

SITE_PACKAGES=$(pipenv --venv)/lib/python3.6/site-packages
DIR=$(pwd)

# Make sure pipenv is good to go
pipenv install

cd $SITE_PACKAGES
zip -r9 $DIR/OldPosts.zip *

cd $DIR
zip -g OldPosts.zip old_posts.py

Makefile

.PHONY: package

package:
	sh package.sh

This should just work™.