levlaz

Posts tagged: hacking

Dockerized PostgreSQL and Django for Local Development

2017-10-31 03:47:38 ][ Tags: hacking python docker django

Docker and docker-compose make it dead simple to avoid dependency hell and have a consistent environment for your whole team while doing local development. This post walks through setting up a new Django project from scratch to use Docker and docker-compose. It is modeled after a previous post that I wrote about doing a similar thing with Laravel and MySQL.

Dockerfile

Nothing too interesting happening here. Installing python and pip.

FROM ubuntu:16.04

# system update
RUN apt update
RUN apt upgrade -y

# python deps
RUN apt install -y python3-dev python3-pip

docker-compose.yml

version: '2'
services:
  app:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    working_dir: /app
    command: bash -c "pip3 install -r requirements.txt && python3 manage.py migrate && python3 manage.py runserver 0:8000"
    depends_on:
      - db
  db:
    image: postgres:9.6.5-alpine
    environment:
      - POSTGRES_USER=feedread
      - POSTGRES_PASSWORD=feedread
    volumes:
      - ./data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

With this in place you can start your Django app with docker-compose up. Each time the app starts it will install the latest dependencies, run migrations, and start serving the app on localhost:8000

Notes

  1. In order to do stuff with the database locally you should add the following record to your local /etc/hosts file

    # /etc/hosts
    
    127.0.0.1 db
    
  2. Since we define - .:/app as a volume, this means that all of your local changes are immediately visible in the dockerized app.

  3. If you need to access the running app or db container you can do so with docker-compose exec app bash or docker-compose exec db bash.
  4. This docker-compose file is not really suitable for production since it is not likely that you would want to build the container each time the app starts or automatically run migrations.
  5. You can add additional services like memcached, a mail server, an app server, a queue, etc., using the same method that we are using above with our database.

I Want to Become a Core Python Developer

2017-10-30 22:14:47 ][ Tags: hacking python

I've been tinkering with python for almost five years now. I am absolutely in love with the language. My new goal is to make enough contributions to the project to join the core team.

This post is my attempt to keep a list of all that I've done in this endeavor. I will keep this up to date on a monthly basis.

Short Term Goals

November 2017

Code

Community

October 2017

Code

Community

Python Mocks Test Helpers

2017-10-27 03:16:06 ][ Tags: hacking python

I've been writing a python wrapper for the CircleCI API over the last week. I wanted to do this "the right way" with test driven development.

I have a couple integration tests that actually hit the CircleCI API, but most of the unit tests so far are using MagicMock to ensure that the basic functions are working as expected.

This generally involves the tedious process of dumping out JSON, saving it to a file, and then reloading that file later on to actually test it.

I wrote two helper functions that make this process slightly less tedious.

Load Mock

The first is a function that loads a file and overrides every request to return that file (typically as JSON).

    def loadMock(self, filename):
        """helper function to open mock responses"""
        filename = 'tests/mocks/{0}'.format(filename)

        with open(filename, 'r') as f:
            self.c._request = MagicMock(return_value=f.read())

Test Helper

The second is a function that runs a real request for the first time and dumps the output to a file.

    def test_helper(self):
        resp = self.c.add_circle_key()
        print(resp)
        with open('tests/mocks/mock_add_circle_key_response', 'w') as f:
             json.dump(resp, f)

Naming it test_helper allows it to be picked up and ran when you run your test suite since by default unittest will capture any methods that start with test.

Usage

An actual example is shown below.

    def test_clear_cache(self):
        self.loadMock('mock_clear_cache_response')
        resp = json.loads(self.c.clear_cache('levlaz', 'circleci-sandbox'))

        self.assertEqual('build dependency caches deleted', resp['status'])

Writing the tests is easy, we just copy and paste the name of the file that was created with test_helper and verify that the contents are what we expect them to be.

This approach has been working very well for me so far. One thing to keep in mind with writing these types of tests is that you should also include some general integration tests against the API that you are working with. This way you can catch any regressions with your library in the event that the API changes in any way. However, as a basic sanity check mocking these requests is a good practice and less prone to flakiness.

Spring Security, Webjars, and MIME type error

2017-05-22 19:45:29 ][ Tags: hacking java

I volunteered to be JLO (Java Language Owner) at CircleCI and I am currently working on getting a sample Spring Framework project running on CircleCI 2.0. I made a simple app bootstrapped with the Spring Initializer. I included Spring Security for the first time and I decided to try out WebJars for static Javascript libraries such as bootstrap. I am using Thymeleaf for templating.The app does not actually do anything yet but I ran into a pretty strange issue today that I wanted to write up here. My home page is pretty straightforward.

<!DOCTYPE html>
<html xmlns:th="http://www.thymeleaf.org">
<head>
    <title>CircleCI Spring Demo</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />

    <link rel="stylesheet" th:href="@{/webjars/bootstrap/3.3.7/css/bootstrap.min.css}" />

    <link rel="stylesheet" th:href="@{/css/style.css}" href="../static/css/style.css" />
</head>
<body>

    <nav class="navbar">
        <div class="container">
            <div class="navbar-header">
                <a class="navbar-brand" href="#">CircleCI Demo Spring</a>
            </div>
            <div id="navbar" class="collapse navbar-collapse">
                <ul class="nav navbar-nav">
                    <li class="active"><a href="#">Home</a></li>
                    <li><a href="#">About</a></li>
                </ul>
            </div>
        </div>
    </nav>

    <div class="container">
        <h1> CircleCI Spring Demo </h1>
    </div>

    <script th:src="@{/webjars/bootstrap/3.3.7/js/bootstrap.min.js}"></script>
</body>
</html>

However, when I tried to load up the app with mvn spring-boot:run none of the styles showed up and console showed the following error message:

Resource interpreted as Stylesheet but transferred with MIME type text/html

It turns out, that a default spring-security config will basically block any request unless you whitelist it. The MIME type is a red herring since what is actually happening is that my spring-security config is redirecting all unauthenticated users to my login page (which is login.html) instead of serving up the stylesheet from the /webjars directory. The solution is to update my security configuration to whitelist anything that comes from /webjars

package com.circleci.demojavaspring;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;

@Configuration
@EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
            .authorizeRequests()
                .antMatchers("/", "/home", "/webjars/**").permitAll()
                .anyRequest().authenticated()
                .and()
            .formLogin()
                .loginPage("/login")
                .permitAll()
                .and()
            .logout()
                .permitAll();
    }
}

Now, the styles load as expected.

The Best Autotools tutorial from the Ajunta docs

2017-05-11 19:15:02 ][ Tags: hacking gnu autotools

Autotools is probably the most overwhelming piece of software that I have encountered. Just when I think that I get how it works, something goes wrong and I spend hours digging through man pages and info docs trying to figure out what is going on.

I wish I had found the Anjuta docs that describe how the "magic" behind their project wizards actually work earlier. Like most IDE's Anjuta takes care of doing a lot of heavy lifting in the background. Unlike most IDE's they have excellent documentation (in addition to the source code of course) on how everything actually works. This is extremely valuable and I am grateful to the folks from the Anjuta project who took the time to describe how all of this works from doing everything as a single line gcc command all the way to clicking buttons via the new project wizard.

If you are new to autotools like me, check out this doc. If you dont care about auto tools but want to see what excellent documentation looks like, check out this doc as well.

Terminal Reader Mode with Pandoc and Less

2017-05-06 09:01:01 ][ Tags: hacking terminal

The other day Aosheng send me an article to read from the verge. When I tried to read it, it took about 5 minutes to load because of the 15 various JavaScript things that were running in addition to ads loading in the background. Firefox was unhappy, and even when I tried to turn on "Reader View" (which strips out all of the junk) it took another minute to load. I've been on a UNIX binge lately so I figured there had to be a clever hack to make my own reader view in a terminal. This is where pandoc comes to the rescue. I've written about this tool in the past discussing how to easily convert Markdown to PDF. It turns out that pandoc also supports arbitrary URL arguments which means that you can convert HTML files on the fly without having to download them first. This means that we can take an arbitrary URL, pass it into pandoc, and spit out plain text. Furthermore, we can pipe this into less to get a nice pager for longer documents. The full string is shown below:

pandoc -f html -t plain
https://www.theverge.com/2017/5/4/15547314/edward-snowden-cory-doctorow-nypl-talk-walkaway
| less

In the example above, -f specifies the input filetype, in this case HTML. -t specifies the conversion filetype, in this case plain text. Pandoc supports a ton of different formats, you can read the man page for more info.

The next logical step is to make a script like my wordpress mutt poster to make this even easier. You could make a simple program called reader and put it in /usr/local/bin/reader. The contents of this script are:

1
2
3
4
5
6
#!/bin/bash
# Terminal Reader Mode using Pandoc and Less

url="$1"

pandoc -f html -t plain $url | less

You can then use this  by typing reader $URL.

Posting to Wordpress via Email with Mutt

2017-05-05 00:27:29 ][ Tags: hacking terminal

Soemtimes you are hanging out in your terminal and you just want to be able to post something to your blog quickly. I was pretty inspired by Derek Siver's OpenBSD post [1] where he really embraces the unix philosophy of having one tool to do a job correctly and putting together various small tools like this to come up with a solution for the problem that you are trying to solve with your computer. Wordpress with Jetpack makes it dead simple to post to your blog via email [2], even if you do not have a mail server configured. I was able to write a three line bash script to "automate" creating a new post from my command line.

1
2
3
4
5
6
7
#!/bin/bash
# Bash Utility to Post to Wordpress using Mutt

subject="$1"
WP_ADDRESS=

mutt -s "${subject}" $WP_ADDRESS

I saved this file in /usr/local/bin/wp and whenever I am inspired to fire off some quick thoughts to this blog I can run wp "Blog Post Title" which dumps me into a vim buffer that once I complete is sent off via mutt to wordpress. [1] https://sivers.org/openbsd [2] https://jetpack.com/support/post-by-email/#examples

Reading gz files with zcat

2017-05-04 23:15:44 ][ Tags: hacking gnu terminal

The Debian Policy Manual dictates that all packages should come with documentation. In order to save space in the debian archive these documents need to be compressed with gzip. There are a ton of these files floating around in the /usr/share/doc directory. Recently I wanted to read some of the documentation. If you try to open the file with cat it spits out binary gibberish. You can of course unzip the file as you normally would and open it up that way, but it turns out there is an easier way. Using zcat you can read the contents of compressed files just like you would with cat.

zcat is identical to gunzip -c. (On some systems, zcat may be installed as gzcat to preserve the original link to compress.) zcat uncompresses either a list of files on the command line or its standard input and writes the uncompressed data on standard output. zcat will uncompress files that have the correct magic number whether they have a .gz suffix or not. GZIP(1) man page.

By default, this will put all of the output into your terminal window, which is fine for most files. The other place where this can come in handy is when you are trying to look through compressed log files. In this case, having to scroll around the terminal may not be a great option. You can pipe the output of zcat into other programs such as less in order to be able to page through long files. For example, if I wanted to read the first 10 lines of a compressed log file, I could do so with the following command:

levlaz@debvm:/var/log$ sudo zcat syslog.2.gz | head -n 10

The output of this command would look like this:

May  2 22:27:43 debvm rsyslogd: [origin software="rsyslogd" swVersion="8.4.2" x-pid="585" x-info="http://www.rsyslog.com"] start
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1a] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1b] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1c] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1d] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1e] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x1f] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x20] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x21] high edge lint[0x1])
May  2 22:27:43 debvm kernel: [    0.000000] ACPI: LAPIC_NMI (acpi_id[0x22] high edge lint[0x1])

Serving a Static Home Page in Rails

2017-04-17 12:00:05 ][ Tags: hacking rails

TIL while working on this issue that if you dump a file into public/index.html then rails will just serve that up for you. A lot of tutorials out there talk about making a static pages controller, which seems like overkill (unless you have a lot of static pages .. but if thats the case why are you using Rails?)

Do not Install Karma Globally

2017-04-15 12:00:07 ][ Tags: hacking testing javascript

Wow, I spent so long trying to figure out why the hell karma was not working for me, it turns out its because it was installed globally. For instance. In my projects package.json I had:

"scripts": { "test": "karma start karma.conf.js" } ...

When I ran npm test - it told me sh 1: karma not found Every other possible combination also did the same thing. i.e.

node_modules/karma/bin/karma

./node_modules/karma/bin/karma

node ./node_modules/karma/bin/karma

I could totally execute this myself from the shell, so I had no idea what was wrong. Then I finally stumbled upon this GitHub Issue. After uninstalling karma globally, npm uninstall -g karma I was able to run npm test without any issues. I still have no idea why this works or didn't work. But at this point I just want to go back to writing tests.

Injecting Stuff into your Python Path

2017-04-13 12:00:18 ][ Tags: hacking python

Similar to a previous post where I wrote about how to run flask tests without installing your app, another common thing that you might want to be able to do is import your app from some arbitrary script. This is especially useful when running your app with apache mod_wsgi. This module expects the app to be installed globally or at least in the python path. Unless you install the app in a traditional sense this will not be true. The solution is just to inject the path prior to running your import statement like this.

sys.path.insert(0, '/var/www/blog')
from blog import app as application

This import will actually work.

Using the Flask CLI

2017-04-11 12:00:24 ][ Tags: hacking python flask

Who knew that flask had a cli? Previously I used to just use manage.py just like Django does it to "do stuff". The CLI is great, but again it follows the theme of kind of wanting you to install your flask app. (I really should do this). So in order to get your app to work you must point the FLASK_APP variable to the actual python file (not your app module). This is true even if you have a true python module. For instance.

export FLASK_APP = blog/blog.py
flask run

Works, while

export FLASK_APP = blog
flask run

Does not. Even though blog consists of:

blog/
  __init__.py
  blog.py

Using Font Awesome with Laravel

2017-04-09 12:00:04 ][ Tags: hacking php laravel

For some reason there is a whole thread on this seemingly simple tasks. In a bootstrapped Laravel 5.4 instance the following worked for me.

Install Font Awesome with NPM

npm install font-awesome

Import font-awesome in your app.scss file

// resources/assets/sass/app.scss // Font Awesome @import "node_modules/font-awesome/scss/font-awesome";

Copy the fonts to public directory

Adding the following to your elixir config in the gulpfile

.copy('node_modules/font-awesome/fonts', 'public/fonts')

My complete gulpfile looks like this:

const elixir = require('laravel-elixir'); require('laravel-elixir-vue-2'); /* |-------------------------------------------------------------------------- | Elixir Asset Management |-------------------------------------------------------------------------- | | Elixir provides a clean, fluent API for defining some basic Gulp tasks | for your Laravel application. By default, we are compiling the Sass | file for your application as well as publishing vendor resources. | */ elixir((mix) => { mix.sass('app.scss') .copy('node_modules/font-awesome/fonts', 'public/fonts') .webpack('app.js'); });

Run Gulp

If you run gulp you should be able to now start using font-awesome everywhere in your app.

Dockerized Laravel and MySQL for local development

2017-04-08 17:49:15 ][ Tags: hacking php laravel docker

Docker is awesome. Its also quite useful for local development. The following Dockerfile and docker-compose.yml will be helpful if you want to do laravel development inside of docker. I am using Ubuntu as a base, but you can probably use the official PHP image as well. Dockerfile

FROM ubuntu:16.04

RUN apt update
RUN apt install -y php7.0 php7.0-zip php7.0-mbstring phpunit curl php7.0-mysql

RUN curl -sS https://getcomposer.org/installer | php 
RUN mv composer.phar /usr/local/bin/composer

RUN composer global require "laravel/installer"

RUN export PATH=$HOME/.config/composer/vendor/bin:$PATH

docker-compose.yml

version: '2'
services:
  app:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/code
    env_file: .env
    working_dir: /code
    command: bash -c 'php artisan migrate && php artisan serve --host 0.0.0.0'
    depends_on:
      - db
  db:
    image: "mysql:5.7"
    environment:
      - MYSQL_ROOT_PASSWORD=password
      - MYSQL_DATABASE=$your_db
      - MYSQL_USER=$your_db_user
      - MYSQL_PASSWORD=$your_db_password
    volumes:
      - ./data/:/var/lib/mysql
    ports:
      - "3306:3306"

.env file Your .env file is what Laravel uses when it starts up set up various things. The only real thing to change is your DB connection info. A full sample is shown below:

APP_ENV=local
APP_KEY=$your_app_key
APP_DEBUG=true
APP_LOG_LEVEL=debug
APP_URL=http://localhost

DB_CONNECTION=mysql
DB_HOST=db
DB_PORT=3306
DB_DATABASE=$your_db
DB_USERNAME=$your_db_user
DB_PASSWORD=$your_db_password

BROADCAST_DRIVER=log
CACHE_DRIVER=file
SESSION_DRIVER=file
QUEUE_DRIVER=sync

REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379

MAIL_DRIVER=smtp
MAIL_HOST=mailtrap.io
MAIL_PORT=2525
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ENCRYPTION=null

PUSHER_APP_ID=
PUSHER_KEY=
PUSHER_SECRET=

Gotchas

  1. In order to do stuff with the database you should add the following record to your local /etc/hosts file

    # /etc/hosts
    
    127.0.0.1 db
    
  2. You should still install npm and run npm install from your local machine so that you can do frontend stuff.

  3. Since we define - .:/code as a volume, this means that all of your local changes are immediately visible in the dockerized app.
  4. If you need to access the running app or db container you can do so with docker-compose run app bash or docker-compose run db bash

Running Flask Tests without installing the app

2017-04-08 12:00:06 ][ Tags: hacking testing python flask

The Flask Docs have a great section for testing. However they assume that you have the app installed with pip install -e . which I almost never do. (Maybe I should start?) I have had trouble with this approach. So one little hack to inject your app into the searchable python path is to do something like this:

export BLOG_PATH=$(pwd) && python tests/blog_tests.py

We use \$(pwd) so that this will "just work" no matter which computer you run it on (i.e. test, dev, other persons dev) Then in your tests/blog_tests.py file you import your flask app like this:

sys.path.insert(0, os.environ.get('BLOG_PATH'))
from blog.blog import app, init_db

You can see the full details here: https://circleci.com/gh/levlaz/blog My biggest issue with installing the app in a traditional sense is that when I run it with apache mod_wsgi it does not seem to actually copy over the right folders and apache is not able to find templates or static files. So that sucks, and I am probably doing something wrong. But this approach works more or less no matter what so sometimes a hack is better than derping around with no solution (as I did for six hours yesterday).

Standard Notes is a Better Project than Braindump

2017-01-17 23:18:14 ][ Tags: hacking python braindump

I released braindump to the world last year to much fanfare. After the initial excitement from being on HN died down, and the PR’s stopped rolling in, it became a personal project once again with very few users. Over the last few weeks I have made several attempts to fix the spaghetti mess that is the current code base by refactoring the current Flask implementation, then rewriting it completely in Django, and even started a branch to investigate rewriting the whole app in PHP using Laravel. Other commitments took precedence and Braindump remains in a fairly usable but not that special state.

Today on HN I read about a new project called Standard Notes which is the most exciting note related project that I have seen in a long time. It solves so many of the problems around cross platform compatibility that plague many other note tools. In addition its goals are to create a standard file format for simple, secure, and durable notes. Even more it has already created a platform, an ecosystem, that allows anyone to come and create additional applications, plugins, and use cases for notes.

These are some of the problems that I set out to tackle when I started braindump. After reading about Standard Notes, and using it for a few hours, I have decided that my time would be better spent contributing to that project instead of continuing to work on Braindump.

Working on Braindump has been amazing. I learned a ton, became a better programmer, and most of all had a lot of fun. I want to thank everyone who tried it, provided feedback, and sent patches. The source code for braindump will remain on GitHub but I would encourage you to try and contribute to the Standard Notes project along with me.

Outstanding Tutorial on writing GTK+ 3 GUI Applications with Python

2016-12-12 09:00:38 ][ Tags: hacking gtk python

GUI programming has always been a black art to me. The idea of event-driven applications with an endless infinite loop listening for events boggles my mind a bit and runs counter to my traditional understand of what algorithms look like.  This is something that I struggle the most with Javascript (which has the benefit of having most of the details abstracted away by the browser), and something that I have never been able to quite grasp using traditional GUI tools like GTK, Qt, and even .NET.

The trend these days is to just use Electron. It seems like more and more apps (Slack, Wordpress, Ghost, Postman, Visual Studio Code, etc.) are drinking the cross platform cool aid. I read this article which discusses some of the negative aspects of choosing Electron as a GUI framework. Drew makes some great points and the following paragraph inspired me to once again take a stab at actually learning a proper GUI framework.

Learn how to use GTK or Qt. Maybe Xwt is more up your alley. How about GNOME’s Vala thing? Learn another programming language. Learn Python or C/C++ or C#. Fun fact: it’ll make your JavaScript better, and once you have it in your toolbox you can make more educated decisions on the appropriate tool to use when you face your next problem.

Source: Electron considered harmful - Drew Devault's Blog

PyGObject (aka PyGI) is the new way of developing GTK+ 3 GUI applications in python. On Ubuntu, installing the python3-gi package is enough to get started making your application "do something". I found an excellent tutorial on using PyGI which does a great job explaining both basic and advanced concepts.

This tutorial gives an introduction to writing GTK+ 3 applications in Python.

Source: The Python GTK+ 3 Tutorial — Python GTK+ 3 Tutorial 3.4 documentation

In addition, this tutorial also has a section on how to use Glade, which is a GUI tool for building GUIs. For any substantial project hard coding the UI is going to get old pretty quickly.

The Python GObject Introspection API is massive. It solves a lot of common problems and also lets you work with a lot of existing GNOME applications. Looking forward to making something useful soon.

Getting Started with Laravel on Ubuntu

2016-11-30 09:00:56 ][ Tags: hacking php laravel

I've really been digging Laravel lately. Especially due to the wonderful documentation and amazing resources provided by Laracasts. Below are some notes on getting going on a local Ubuntu install. I am running Ubuntu 16.10 and these notes assume a fresh install.

Install PHP 7.0 and additional dependencies

sudo apt install php7.0 php7.0-zip php7.0-mbstring phpunit

Install Composer (Globally)

curl -sS https://getcomposer.org/installer | php
sudo mv composer.phar /usr/local/bin/composer

Install the Laravel CLI (Globally)

composer global require "laravel/installer"

Add globally installed composer commands to the PATH

Add the following to the end of your ~/.bashrc file

# Add Composer to the PATH
export PATH=$HOME/.config/composer/vendor/bin:$PATH

You can either source the ~/.bashrc file or open a new terminal window.

Verify everything works

You can do this by running laravel new test_project. Then go to the directory where the new test_project is created with cd test_project. Once you are in the new project directory install all local dependencies with composer install and then run it with php artisan serve. I ran into an issue right away that had to do with the APP_KEY. The error manifested itself as:

The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths.

A great explanation is shown here but the steps to get a fully functional base install going are:

# Copy the .env.example file to .env
cp .env.example .env

# Generate App Key
php artisan key:generate

Now if you run php artisan serve you will see a fully running Laravel app. Happy Hacking!

Whatever hacky script you are writing already exists in GNU Core Utilities

2016-11-16 23:52:10 ][ Tags: hacking gnu bash

When I think of bash, I think of writing hacky scripts that do random things utilizing "bash commands". It turns out that the parts of bash that "do stuff" such as echo, cut, cat are part of a larger program called GNU Core Utilities.

The GNU Core Utilities are the basic file, shell and text manipulation utilities of the GNU operating system.These are the core utilities which are expected to exist on every operating system.

Source: Coreutils - GNU core utilities I am working on a general purpose backup utility and this evening I was moments away from writing something like this: perl -e (print split("/\//", "/foo/bar/baz.tar.gz") My goal was to try to extract the base file name from a given directory (I recognize that that code does not actually do that). Then I realized that this was pure madness and there had to be a better way. This is when I discovered the handy basename program. It simply does the needful. GNU Core Utilities is full of all sorts of gems such as this one. My main takeaway from this is to read the entire GNU Core Utilities manual so I can stop writing horrible things.

That UI Bug with Missing Data is a Security Issue

2016-11-16 20:47:06 ][ Tags: hacking python braindump security

This is (sometimes) a development blog, so I am going to write about some failed development of mine since writing about success is much less interesting. You know that UI bug that someone added to your GitHub Issues where there is some missing data? You know the one, it only happens in production, all of your tests pass, and you marked it as a low priority. Yeah, that one. It's probably a security bug and you should look into it right away. At least, that is the lesson I taught  myself yet again when I began to research this bug. Before

<a id="shared_note.id)" href=""{{">
{{current_user.notes.filter_by(id=shared_note.id).first().title}}
</a>

After

<a href="{{ url_for('main.note', id=shared_note.note_id) }}">
{{current_user.notes.filter_by(id=shared_note.note_id).first().title}}
</a>

The difference is very subtle, but the key issue here is shared_note.id vs shared_note.note_id; I released a feature a few weeks ago that showed you all of the notes that you have shared. Locally everything worked fine, but I noticed later on, once it was in production, that the note title was not showing up. This is, of course, due to the fact that rather than showing the title of the note with the ID shared_note.note_id (the foreign key linking to the note) I was showing the title for the note with the primary key of shared_note.id. The reason why this is a security issue is because this allows someone to share a bunch of notes and start seeing the titles for notes that they do not own. The reason why this worked locally is because I am only testing with a single user, with a single notebook, with a single note, and with a single shared note. This means that in this specific case all of the Primary Keys and Foreign Keys are usually "1" so everything just happens to work.

Key Takeaways

Laravel Homestead on Ubuntu 16.04

2016-11-15 18:23:37 ][ Tags: hacking php laravel

Yesterday I wrote about my first steps with Laravel and Homestead and complained about how I had some trouble getting started. It turns out this has nothing to do with Laravel and instead has to do with a Vagrant bug in the version that is shipped by default with Ubuntu 16.04 LTS.

vagrant up throws The following SSH command responded with a non-zero exit status error for Laravel Homestead

Source: 16.04 - vagrant up throws The following SSH command responded with a non-zero exit status error for Laravel Homestead - Ask Ubuntu

Installing Vagrant manually did the trick and everything worked like a charm. I am so impressed so far, the bootstrapped Laravel project is elegant, well structured, and beautiful by default. I am almost afraid to write any code because I will probably ruin it.

First Steps with Laravel and Homestead

2016-11-14 21:50:20 ][ Tags: hacking php laravel

I'm checking out Laravel for the first time. I think they have a very nice web site, but slightly confusing documentation. First I tried to get composer working but the whole experience felt weird. Once I got it sorted out, I had some other issues with my local environment so then I went ahead and started to read about Homestead.

Laravel strives to make the entire PHP development experience delightful, including your local development environment. Vagrant provides a simple, elegant way to manage and provision Virtual Machines.

Source: Laravel Homestead - Laravel - The PHP Framework For Web Artisans

This was a bit more promising, since Vagrant makes everything dead simple. I was a bit surprised that the docs suggested creating a single Homestead box and sharing it across all of your Laravel projects since this is a practice that I have not seen before.

I went ahead and did what I usually do and create a Vagrant box for my specific project. It's been many years since I have written any "real" PHP, looking forward to seeing what Laravel has to offer.

Development on Windows

2016-10-27 19:03:35 ][ Tags: hacking windows

Last weekend CircleCI hosted ClojureBridge and I volunteered as a TA. It was a super rewarding experience and I hope to be more involved in these types of events in the future. One thing I noticed (and I have noticed a similar trend in my previous experience as a mentor at various hackathons) is that many students and junior developers run windows.

Trying to run and develop Node, PHP, Rails, and even some Python apps on windows can be a pretty painful experience. Most developers who write in these languages use either Linux or OS X, and nearly all production applications written in these languages are running on a Linux server. In addition a lot of documentation assumes that you are running either Linux or OS X for many libraries and sample projects.

Most commonly Windows users will be encouraged to use a VM with VirtualBox and Vagrant. This is not bad advice and for most folks will probably be the best solution while you are learning. Vagrant is certainly worth learning, and I cannot stress how important it is to have a solid understanding of Linux.

However, I am stubborn enough to try to get things to work on Windows and after last weekend I consider myself a self proclaimed Clojure on Windows expert.

Common Gotchas

  1. The Windows PATH is very different from Linux or OS X.
  2. Remember that file paths in Windows use “\” while in Linux and OS X they use “/”
  3. CMD, vs PowerShell, vs Git Bash, vs Bash For Windows. Common theme is differences in the PATH. If you are used to doing stuff via the CLI, or are following some instructions online you can not typically just copy and paste.

Although Windows is generally more difficult for general FOSS development, when it comes to installing third party services and tools (such as databases, queues, app servers) it is typically easier from a beginners perspective to get started because pretty much every major service out there has a point and click GUI installer and configuration system.

There are certainly pros and cons to local windows development. I have a pretty beefy machine at home that I used to play games sometimes. I installed some development tools on it and realize that its a much more powerful computer than my Macbook Pro. I look forward to finding the edge cases and complaining about them in public here on this blog.

Editing the Windows PATH

2016-10-26 05:13:52 ][ Tags: hacking windows

As a developer you will often install random things onto your computer and hope that they magically work when you execute them from a command prompt. Most package managers on Linux, BSD, Unix, and will install things that are already in your PATH so life is simple. On Windows its slightly a different story.

Just like in the other operating systems listed above you can add any arbitrary directory to your windows PATH. The strategy that I came up with was making a folder called C:\DevTools and then any time I need to install any library or command line utility (like nuget) that needs to be in my path, I will simply plop it in that folder and everything will just work.

The detailed process is outlined below. In this guide, I am assuming that you want to follow my convention of using a folder called C:\DevTools for all the things. Keep in mind you can add anything that you wish to your PATH using the same steps.

  1. Open up Documents, right click on This PC and  select PropertiesThis PC View
  2. On the left hand side of this Window, select Advanced System SettingsAdvanced System Settings
  3. At the bottom of this window, select Environment Variables...Windows Environment Variables
  4. Select Path and then Select Edit...
  5. Append ;C:\Devtools; to the Path variable and select OK.Editing the Windows PATH
At this point everything inside of this folder will be in your PATH and you will be able to execute anything inside of this folder from the command line.

 

 

Working with NuGet CLI

2016-10-26 04:49:59 ][ Tags: hacking windows nuget

I have been working with .NET Core lately and I am pretty excited about the future of .NET running outside of just windows. Ever since my first tech job at an enterprise healthcare company, I have had a soft spot in my heart for C#. I am also a huge fan of Visual Studio Code compared to the big ball of bloat that is Visual Studio proper, vscode is a refreshing take on a simple IDE.

The one thing that is not as intuitive as I thought was working with NuGet outside of Visual Studio.

NuGet is the package manager for the Microsoft development platform including .NET

Source: NuGet Gallery | Home

The easiest way to install third party libraries and tooling is with Nuget. Buried deep within the documentation is an executable called Nuget CLI that you can download to use nuget completely outside of Visual Studio. This supposedly works on OS X and Linux as well.

The quick and dirty way to get started is:

  1. Download the latest version of the Nuget CLI
  2. Copy it somewhere that you don't mind being in your PATH. I have a special folder called C:\DevTools
  3. Open up a command prompt and run the nuget command

[caption id="attachment_9" align="alignnone" width="450"]Nuget Running in Command Prompt Nuget Running in Command Prompt[/caption]

 

 

I Inherited an Open Source Project

2016-09-13 21:09:37 ][ Tags: hacking

Last week I was browsing Twitter when I noticed a retweet where Honza was putting all of his open source projects up for adoption since he is going to go work at Apple. I took a look at the long list of neat projects that were up for adoption and decided to throw my hat in the ring for Pong.

Pong is a simple self-hosted server monitoring tool written in Swift. I have been increasingly interested in server side swift so this is a great opportunity to dive in a learn some more.

I don't have any major plans for this project right now other than being a good steward and a responsive maintainer. If you have some ideas of things that you would like to see please let me know!

Working with chruby and nvm on Ubuntu

2016-09-03 21:07:17 ][ Tags: hacking ruby node

I am setting up a new Ubuntu 16.04 dev environment. I wanted to take some notes on my progress. I am trying out chruby after it being highly recommended by the ruby gurus Eric and Zach. A natural place to start would be to get this Jekyll blog running locally. I ran into a few stumbling blocks.

Installing chruby and a Ruby version

There are many different ways to do this, I chose the method below.

  1. Download and Install chruby
    wget -O chruby-0.3.9.tar.gz https://github.com/postmodern/chruby/archive/v0.3.9.tar.gz
    tar xf chruby-0.3.9.tar.gz
    cd chruby-0.3.9/
    sudo make install
    
  2. Add chruby to your .bashrc
    source /usr/local/share/chruby/chruby.sh
    source /usr/local/share/chruby/auto.sh
    
  3. Install dependencies
    sudo apt-get install -y build-essential bison zlib1g-dev \
    libyaml-dev libssl-dev libgdbm-dev libreadline-dev \
    libncurses5-dev libffi-dev
    
  4. Install Ruby 2.3.1 This will work for any version of Ruby, but 2.3.1 is the one that I needed.
    wget http://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.1.tar.bz2
    tar xf ruby-2.3.1.tar.bz2
    cd ruby-2.3.1/
    ./configure --prefix=/opt/rubies/ruby-2.3.1
    make
    sudo make install
    
  5. Install Gems for Jekyll Now that we have Ruby installed, we can install bundler and install all of our Gems.
    chruby 2.3.1
    gem install bundle
    bundle install
    

Once everything had been installed I tried to start up jekyll, and of course it did not work since I do not yet have node installed.

/home/levlaz/.gem/ruby/2.3.1/gems/bundler-1.12.5/lib/bundler/runtime.rb:89:in
`rescue in block (2 levels) in require': There was an error while trying to load
the gem 'jekyll-coffeescript'. (Bundler::GemRequireError)
Gem Load Error is: Could not find a JavaScript runtime. See
https://github.com/rails/execjs for a list of available runtimes.

Since I often work with many different versions of node, I took the opportunity to install nvm rather than the version of node that is available in the Ubuntu repositories.

Installing nvm

nvm is a really nice tool for working with various versions of node.

  1. Run the installer script
    curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.31.7/install.sh | bash
    
    This will install nvm and add the appropriate lines to your ~/.bashrc file in order to load properly.

  2. Install the latest version of node

    nvm install node
    
  3. Source NVM To get nvm working in the same terminal, you can run source ~/.bashrc, alternatively you can open a new terminal and nvm will automatically be sourced since the installer script added the appropriate bits for us.

You can now load jekyll with jekyll -s. Not only do we have Jekyll running like a charm, we also are ready to work with any version of Ruby or Node with chruby and nvm.

Show all Flask Routes

2016-08-28 21:02:21 ][ Tags: hacking python flask

One of my favorite things about rails is rake routes. This command will show you all of the routes that your application is currently aware of. I was searching for something similar in Flask and came across this snippet. Sadly, it did not work for me. I am using Python 3, so that may have something to do with it. In any case, I did not have time to dig into what was wrong, because just the other day I read this amazing introduction to functional python where my biggest take away was that as soon as you see multiple for loops, its time to use map. With my functional skills in tow, I wrote a simple one liner to print all of my routes.

@manager.command
def routes():
    import pprint
    pprint.pprint(list(map(lambda x: repr(x), app.url_map.iter_rules())))

Running this on braindump, shows me everything that I need to know.

[levlaz braindump]$ python3 manage.py routes
["<Rule '/auth/change-password' (GET, HEAD, OPTIONS, POST) -> "
'auth.change_password>',
"<Rule '/auth/change-email' (GET, HEAD, OPTIONS, POST) -> "
'auth.change_email_request>',
"<Rule '/auth/unconfirmed' (GET, HEAD, OPTIONS) -> auth.unconfirmed>",
"<Rule '/auth/register' (GET, HEAD, OPTIONS, POST) -> auth.register>",
"<Rule '/auth/confirm' (GET, HEAD, OPTIONS) -> auth.resend_confirmation>",
"<Rule '/auth/logout' (GET, HEAD, OPTIONS) -> auth.logout>",
"<Rule '/auth/login' (GET, HEAD, OPTIONS, POST) -> auth.login>",
"<Rule '/auth/reset' (GET, HEAD, OPTIONS, POST) -> "
'auth.password_reset_request>',
"<Rule '/empty-trash' (GET, HEAD, OPTIONS) -> main.empty_trash>",
"<Rule '/notebooks' (GET, HEAD, OPTIONS, POST) -> main.notebooks>",
"<Rule '/favorites' (GET, HEAD, OPTIONS) -> main.favorites>",
"<Rule '/settings' (GET, HEAD, OPTIONS) -> main.settings>",
"<Rule '/shutdown' (GET, HEAD, OPTIONS) -> main.server_shutdown>",
"<Rule '/archive' (GET, HEAD, OPTIONS) -> main.view_archive>",
"<Rule '/search' (GET, HEAD, OPTIONS) -> main.search>",
"<Rule '/trash' (GET, HEAD, OPTIONS) -> main.trash>",
"<Rule '/add' (GET, HEAD, OPTIONS, POST) -> main.add>",
"<Rule '/' (GET, HEAD, OPTIONS, POST) -> main.index>",
"<Rule '/static/bootstrap/<filename>' (GET, HEAD, OPTIONS) -> "
'bootstrap.static>',
"<Rule '/auth/change-email/<token>' (GET, HEAD, OPTIONS) -> "
'auth.change_email>',
"<Rule '/auth/confirm/<token>' (GET, HEAD, OPTIONS) -> auth.confirm>",
"<Rule '/auth/reset/<token>' (GET, HEAD, OPTIONS, POST) -> "
'auth.password_reset>',
"<Rule '/delete-forever/<id>' (GET, HEAD, OPTIONS, POST) -> "
'main.delete_forever>',
"<Rule '/notebook/<id>' (GET, HEAD, OPTIONS) -> main.notebook>",
"<Rule '/notebook/<id>' (DELETE, OPTIONS) -> main.delete_notebook>",
"<Rule '/favorite/<id>' (GET, HEAD, OPTIONS, POST) -> main.favorite>",
"<Rule '/restore/<id>' (GET, HEAD, OPTIONS, POST) -> main.restore>",
"<Rule '/archive/<id>' (GET, HEAD, OPTIONS) -> main.archive>",
"<Rule '/delete/<id>' (GET, HEAD, OPTIONS, POST) -> main.delete>",
"<Rule '/static/<filename>' (GET, HEAD, OPTIONS) -> static>",
"<Rule '/share/<id>' (GET, HEAD, OPTIONS, POST) -> main.share>",
"<Rule '/note/<id>' (GET, HEAD, OPTIONS) -> main.note>",
"<Rule '/edit/<id>' (PUT, OPTIONS) -> main.edit>",
"<Rule '/tag/<name>' (GET, HEAD, OPTIONS) -> main.tag>"]

Now, all that is left is for me to figure out why my API routes are missing. :)

Javascript ES6

2016-08-02 19:10:25 ][ Tags: hacking javascript

I've spent more time in the last few weeks learning more Javascript. This has resulted in a whole bunch of random inconsistent apps, hacks, and scripts floating around GitHub authored by me. My go to resource whenever I want to get a better idea of how something works and what "good" Javascript looks like is Todo MVC. This is a great site that provides tons of sample code in essentially every flavor of Javascript. I was especially excited to see that they updated the examples in their repo to include ES6 examples in Vanilla JS. Another great general resource for me has been the Mozilla Developer Network site. Which provides really nice documentation and code snippets.

Starting to write Javascript today can be pretty overwhelming. There are so many different dialects, frameworks, tools, and practices. I like that ES6 took a bunch of the things that were missing from the previous versions of JS and made them more standardized. I hope that in the future we will be able to shed away some of the tooling around writing sane javascript since it will be included in the language (and more importantly supported on server and browsers) natively.

One of the most frustrating things is that a lot of the samples and StackOverflow posts that you see are riddled with anti patterns. So trying to find the proper way to do something can be a bit challenging. I hope that by continuing to standardize JS and develop best practices that some of the less desirable solutions will stop making the front page on search results.

I have been babeling, grunting, gulping, bowering, and package.jsonsing all month. It's actually kind of fun, I feel more confident being able to reason about how JS fits into the overall picture of my application and also how to structure more complex applications.

Recreating Foreign Keys with Alembic

2016-05-30 18:59:18 ][ Tags: hacking python databases

Alembic is a great tool for keeping track of schema changes in python applications. I am using it to manage DB migrations for braindump along with Flask SQL Alchemy as my ORM. One challenge is managing proper foreign key constraints. By default if you define a foreign key relationship in your schema definition it will not generate the proper migration code. For example, in braindump we have a one to many relationship between users and notes.

class User(UserMixin, db.Model):
    __tablename__ = 'users'
    id = db.Column(db.Integer, primary_key=True)
    email = db.Column(db.String(254), unique=True, index=True)
    password_hash = db.Column(db.String(256))
    confirmed = db.Column(db.Boolean, default=False)
    avatar_hash = db.Column(db.String(32))
    created_date = db.Column(db.DateTime(), default=datetime.utcnow)
    updated_date = db.Column(db.DateTime(), default=datetime.utcnow)

    notes = db.relationship(
        'Note', backref='author',
        lazy='dynamic', cascade="all, delete-orphan")

Even though we define the cascade behavior using SQLAlchemy. When we generate the migration with alembic we get something like this:

sa.ForeignKeyConstraint(['author_id'], ['users.id'], ),

Notice how we are missing the ondelete action. What we actually want is something like this:

sa.ForeignKeyConstraint(['author_id'], ['users.id'], ondelete='CASCADE')

Running the default migration will not create the proper relationship in your database and in our case we are not able to delete a user until we have deleted all of the related records as well. There are two ways to fix this. If you catch this before running your migration adding ondelete='CASCADE' will create the proper relationship. If you are like me, and do not catch this, then you will need to run a second migration to remove and recreate these keys. The migration code to do this is shown below:

from alembic import op
import sqlalchemy as sa

def upgrade():
    with op.batch_alter_table("notes") as batch_op:
        batch_op.drop_constraint(
            "notes_author_id_fkey", type_="foreignkey")
    op.create_foreign_key(
        "notes_author_id_fkey", "notes", "users",
        ["author_id"], ["id"], ondelete="CASCADE")

Now you have the proper foreign key constraints and the CASCADE action exists in the DB.

My First Pip Package

2016-04-21 18:50:24 ][ Tags: hacking python

I finally got around to packaging up pg2cf and I am excited to see it out in the wild on pypi. Python packaging is pretty straightforward and this makes distribution of this tool much easier for us internally.

I could not have done it without this awesome python packaging guide.

Using setuptools also makes a lot of other things easier. For instance running tests is as simple as python setup.py test and installing the package locally can be done with python setup.py install. The main benifit of course is that now pg2cf is an executable so you once it is installed you can just run it.

The only "gotcha" that I ran into was that setuptools does not support markdown. Which makes it kind of weird since Markdown is now the standard for README and other documentation on GitHub.

I worked around this by using pandoc, this way I am able to convert my README.md to README.rst easily with pandoc README.md -o README.rst and then use that for PyPI.

Installing Leiningen on Windows

2015-09-05 18:24:09 ][ Tags: hacking windows clojure

Leiningen is an awesome tool for getting up and running with Clojure. They have a super simple bash or .bat script for getting up and running on OS X, Linux, and Windows. The home page does an awesome job explaining how to get started with OS X and Linux but not so much on Windows.

This is a very similar process to what I wrote about previously on Setting up ANTLR4 on Windows. If you do not already, make sure that you have Java Installed (the latest version should work fine).

Create Batch Command for lein

Download lein.bat and save it somewhere. I put all of my random Java libraries in C:\Javalib to make them easy to find. So that is where I put this bat script.

Add C:\Javalib to your PATH

  1. Open up File Explorer
  2. Right Click on This PC
  3. Select Properties
  4. On the left hand side of the new window select Advanced System Settings
  5. On the bottom of this screen select Environment Variables…
  6. Find the PATH variable
  7. Append ;C\Javalib\; to the existing PATH variable

Install and Test to make sure everything works

  1. Open up a command prompt and type in lein self-install. This will install lein.
  2. Now you can type lein repl to start a REPL session
user=> (+ 2 2)
;= 4
user=> (printf "Clojure on Windows is Awesome!")
;= Clojure on Windows is Awesome!nil

If nothing is bombing out, you are all set and ready to hack on some Clojure! Why not check out the CircleCI frontend for inspiration?

Hacking an Angular.JS game for Fun and (fake) Profit

2015-07-12 18:12:47 ][ Tags: hacking javascript

Last night I came across an interesting game called Giant Shaft Enterprises. In this game you play the role of a CEO of a fictitious company and the goal is to maximize your profits and reduce costs. You do this by performing business opportunities such as “Network with business leaders”, “Push the envelope”, and “Streamline workforce”. You also can hire employees, purchase buildings, establish various departments, and motivate your workforce with seasoned upper managers. The last thing that you can do in the game is purchase upgrades such as health benefits, electric car charging ports or allow upper managers to hold useless meetings. These upgrades maximize the performance of your workforce. All of these things make more money for your company and allow you to get achievements. This is pretty much like any other point, click, and wait game that you would find in the “Top 10 Free Games” section in an App store, but the difference is that this game does not take any real money from you. The game is very entertaining and kind of trolls the world. For example, buying more cubicles for your company provides a higher ROI than hiring additional upper managers. The game is also strangely addictive, I found myself clicking on the same button thousands of times to get more money and increase the profitability of my company. I must have played for a few hours before I began filling a bit silly for wasting my time clicking on a button. A few hours later not only was I feeling silly I was also beginning to get an repetitive stress injury in my wrist. That is when I realized that clicking on the button to watch my company sore to greatness was not providing me with the highest ROI. Then I decided to streamline my efforts and opened up the Chrome Javascript Console to see if I could outsource me clicking this button to a simple Javascript program. Luckily, this was pretty straightforward. The app is written in Angluar.JS and all of the various buttons that you click to do things are exposed in the DOM. Using JQuery you can access these buttons a simulate a click through the Javascript console that can be found in the Developer Tools of most browsers. For example, the button that lets you take advantage of business opportunities (and instantly raise more money) calls a function called doBusiness. You can select it, and send it a click() like this:

$('div[ng-click="doBusiness(opportunity, $event)"]').click();

You can also send it 10, 100, or 1000 clicks by wrapping this in a loop.

for ( i=0; i < 100; i++ ) { $('div[ng-click="doBusiness(opportunity, $event)"]').click(); }

All of the things that you buy in the game use the same function called buyStoreItem, but the individual items (such as employees, cubicles, and harware) are referenced by index. Using the similar method you can purchase things in various multiples by wrapping the function in a for loop like so:

# Buy Cubicle   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[1].click(); }

I found that if I try to send more than 1000 clicks at once, the app tends to freeze and bomb out. Looking in htop, chrome is using up 100% of my CPU so it looks like 1000 is the magic number. Javascript is fast and it looks like it sends all 1000 clicks pretty much at once which confuses the app and makes it very sad. I realize this goes against the spirit of the game, but nonetheless it was a fun little exercise and games should reward cleverness. A full list of ~~Cheats~~ Synergistic ROI Optimizers is listed below:

# Giant Shaft Cheats   The lines below simulate clicks, to run then open up the Javascript Console (in Chrome this is Ctrl+Shift+i) and enter them. To change the amount of simulated clicks change the second value in the for loop ( i < 100 ) to something other than 100. Going above 1,000 will probably crash the app :)   # Do Business Opportunities   for ( i=0; i < 100; i++ ) { $('div[ng-click="doBusiness(opportunity, $event)"]').click(); }    # Buy Minium Wage Worker   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[0].click(); }   # Buy Cubicle   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[1].click(); }   # Buy Salary Employee   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[2].click(); }   # Buy Hardware   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[3].click(); }   # Buy HR Department   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[4].click(); }   # Buy Accounting Department   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[5].click(); }   # Buy Benefits Package   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[6].click(); }   # Buy (Useless) Upper Management   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[7].click(); }   # Buy Executive   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[8].click(); }   # Buy Office Building   for ( i=0; i < 100; i++ ) { $('div[ng-click="buyStoreItem($event, item)"]')[9].click(); }

Thanks to this little hack I am the king of Giant Shaft Enterprises. Thank you to Eric Freeman for making this awesome game! **UPDATE 1: ** I was curious to see what the max number of clicks I could pass through was. I enabled logging to show me when the loop made the 100th click.

for ( i=0; i < 1000; i++ ) { $('div[ng-click="doBusiness(opportunity, $event)"]').click(); if (i % 100 == 0 ) {console.log('Click #' + i)}}

It looks like the app actually does not crash … but becomes super slow after about 3000 clicks. So to optimize ROI you should only cheat with around 1000 fake clicks at a time. **UPDATE 2: ** The final step of this was obviously to automate the automated clicks. Using Javascripts setInterval method, we can have the script run indefinitely. For example the following code, does 100 business opportunities every 3 seconds, forever.

setInterval(function() {for ( i=0; i < 100; i++ ) { $('div[ng-click="doBusiness(opportunity, $event)"]').click(); if (i % 10 == 0 ) {console.log('Click #' + i)}}}, 3000);

ezBadge v2: Now in Stunning Javascript HD

2015-07-11 17:56:30 ][ Tags: hacking javascript

If you have ever wondered what it looked like to convert an application from Python to Javascript, you are in luck. After finishing ezbadge yesterday, I came to the realization that the entire thing could and should have been written in Javascript. There is absolutely no reason for this thing to send any requests to a server in order to process the URL and render the markdown.

So, I did just that. Introducing, ezBadge v2 After removing all of the Flask, I was able to create two Javascript functions that essentially did the same thing as the Flask app was doing for me.  The added benefit is that other than serving the initial HTTP request, I do not have to handle any additional requests and the entire thing now runs in the browser.

This was a fun little exercise, and I think the Javascript version of this application is a bit more elegant. Rather than appending the template with Jinja as I was doing before, I am now spawning a modal (after doing some form validation) and then using the Marked library to render the preview of the badge markdown. I could have done all of the UI stuff with Flask without any issues, but the fact that the entire thing now runs in the browser saves bandwidth, computing resources, and time.

You can see the transformation for yourself in the ezBadge GitHub repo. In the next iteration of ezBadge I plan on adding more choices for the types of badges that you can make. If you have any suggestions please feel free to leave a comment here or open up a github issue.

Introducing ezBadge

2015-07-10 17:55:51 ][ Tags: hacking python

A few months ago I wrote about my struggles with Markdown Image Links. The entire reason why I was working on that before was to put cute little badges in my Github repos. Today I took this a whole step forward and wrote a small web application that just “does the needful” for you. ezBadge makes it easy to transform a regular Github Repo URL into a beautiful badge by writing the markdown for you and allowing you to just copy and paste it into your README.md file or anywhere else where Markdown is supported. You just feed it a valid Github repo URL and it spits out the Markdown along with a preview of what you will see if you put that markdown somewhere. So far, the only type of Badge that it makes is a CircleCI badge but I plan on adding more in the near future and if there is a badge that you would like to see, please open up an issue for this project.

Module Import Errors in Python when Executing a Program

2015-06-24 17:55:12 ][ Tags: hacking python

I ran into a weird issue today with a python program where even though I have the pyyaml module installed python would complain that ImportError: No module named yaml when I tried to execute my program. I have ran into this in the past and the solutions posted online vary greatly and usually didn’t work for me. I think I have found the root cause of this and wanted to quickly share it here. I am running Python 2.7.9 on OS X Mavericks. Pyyaml was installed via pip and works just fine. The interesting part about this is that when I execute the program by running python program.py everything works fine. If I chmod +x program.py to make it executable and then try to execute it by running ./program.py then this issue begins to arise. The key issue happens to be the header that tells the OS where to find python. When you run python from a shell in OS X it is looking to /usr/local/bin/python to find and execute python. You can verify that this is true by running which python in a terminal. When you execute a python program without explicitly calling python then it will look for python in your file header. The first line of a file typically looks like this #!/usr/bin/python. This was the cause for the descrepancy in my case. Since the python that is called when I say python is different than the python that I defined in my header the module was not able to load. Pip installs all modules wherever your main python is. The fix for this issue is to change the header from #!/usr/bin/python to #!/usr/bin/env python This way it will look to whatever the default python installation is, and in the event that you are running a virtual environment there will not be any clashing. This seems like one of those tricky bash gotcha’s that happen sometimes. Using #!/usr/bin/env python by default is a safe bet to make sure you don’t run into these types of issues.

Getting Started with Arduino Uno on Fedora 21

2015-02-15 02:37:30 ][ Tags: hacking hardware

With Radio Shack going out of business, I was lucky enough to get my hands on all sorts of really neat Arduino gear. I now have an endless supply of random parts, three arduinos, and about two of every shield. I am super excited to get started hacking away! Getting started with the Arduino IDE on Fedora is pretty straightforward but takes a little bit of work outside of yum. There is a version of the arduino IDE available in the default Fedora repo, but this is an older version and sadly it was not able to detect the USB “serial” port by default. Rather than digging around for the solution, I downloaded the latest version of the IDE from the arduino website and was happy to see that it worked perfect out of the box with my Arduino Uno.

Getting the Latest Arduino IDE on Fedora 21

  1. First we will download the latest tar ball of the Arduino IDE. At the time of writing this is version 1.6.0. Open up a terminal and issue the following command:

    `wget http://arduino.cc/download_handler.php?f=/arduino-1.6.0-linux64.tar.xz`
    
  2. Next we will extract this file and move it over to the /opt directory

    `tar -xvf arduino-1.6.0-linux64.tar.xz`   `sudo mv arduino-1.6.0 /opt`
    
  3. Lastly, we will make a symbolic link to alllow us to run the arduino IDE simply by typing the command “arduino”

    `sudo ln -s /opt/arduino-1.6.0/arduino /usr/bin/arduino`
    

Now you should be able to type arduino in a terminal and see the IDE come to life. If your arduino Uno is plugged in you should see it be immediately recognized under the ports. I cant wait to share some of the cool stuff that I build with all of these new toys!

Programming Sockets in Python

2015-01-25 02:35:54 ][ Tags: hacking python

EDIT: I am sorry for derping out of control. When I initially published this post it was called “Programming Web Sockets in Python”, this is just flat out wrong. What we are making here is just a regular socket. Web Sockets and regular sockets are similar but are certainly not the same thing. I hope you will still find this useful! Sockets are pretty much the basis of how applications work on the Internet. Python makes it super easy to get started programming sockets. In this brief introduction we will create a simple server that greets the user when it receives incoming requests from the client application. Due to my recent obsession with Linux Containers we will also be implementing this inside of two containers. Containers make it really simple to simulate a network because you can create additional hosts in seconds.

Creating your Containers

I am running Ubuntu 14.04. So creating two additional containers can be achieved by running the following as the root user.

lxc-create -t download -n pyServer # Choose ubuntu, trusty, amd64 when prompted # Then clone the first container lxc-clone -o pyServer -n pyClient

Running the Server

Now that we have created our containers lets jump into our server container and fire up our simple server application. We can start up the container by issuing the following command as root: lxc-start -n pyServer -d, this will start the container as a daemon. Let’s go ahead and get into by attaching the container. I like to do this inside of screen so that we can easily get in and out of the container. Create a screen session screen -dRR pyServer and once inside the screen attach the container lxc-attach -n pyServer Once we are inside the container we need to install python and launch our simple server.

apt-get install python vim pyServer.py

Inside of vim (or your favorite text editor) we need to enter the following simple python code.

from socket import *  serverPort = 12000  serverSocket = socket(AF_INET, SOCK_DGRAM)  serverSocket.bind(('', serverPort))  print "The server is ready to rock and roll!"  while 1:     name, clientAddress = serverSocket.recvfrom(2048)     response = "Hello " + str(name) + "! You are really good at socket programming"     serverSocket.sendto(response, clientAddress)

The code should be pretty straightforward. We are creating a new serverSocket that is bound to port 12000. When it receives requests (which include a name) it responds with an encouraging message. Fire up this server by running python pyServer.py if all goes well you should see a message that states This server is ready to rock and roll! Exit the container (and the screen session) by pressing Ctrl+a and Ctrl+d

Running the Client

Now that we have our server up and running, lets get our client working as well. Before we move forward, lets grab the IP address of our server container because we will need it soon. You can get the IP by running lxc-ls --fancy. Launch the client container, attach it in screen, and install python in the same way that we did previously.

lxc-start -n pyClient -d screen -dRR pyClient lxc-attach -n pyClient apt-get install python vim pyClient.py

In vim, lets create the pyClient.py program by entering the following code.

from socket import *  # Replace the IP address in serverName with the IP of your container that you      grabbed previously. serverName = '10.0.3.211' serverPort = 12000 clientSocket = socket(AF_INET, SOCK_DGRAM)  name = raw_input('Please enter your name:')  clientSocket.sendto(name, (serverName, serverPort)) response, serverAddress = clientSocket.recvfrom(2048) print response clientSocket.close()

This code is also pretty straightforward. It asks the user for their name, sends it to the server, and prints the response. You can try this out now! Save the file and execute your python program by running python pyClient.py. After entering your name and pressing enter you should see a response from your server with the encouraging message. This was a pretty trivial exercise, but we can quickly see that we can expand upon this basic code to create much more interesting and complex applications. We can also leverage the power and simplicity of LXC to create a simulated large network for distributed applications.

Export Pretty Code from SublimeText

2015-01-04 01:40:56 ][ Tags: hacking

There is this awesome plugin for Sublime Text called Sublime Highlight that allows you to export your syntax highlighted code directly into other applications. This comes in handy if you are using a word processor or presentation software to share some code examples.

Diagraming Tools for Linux

2014-12-21 01:37:58 ][ Tags: hacking

In Grad school and supposedly in the real world, we make a lot of diagrams of stuff. There are a lot of tools to do this. For Windows, Visio probably works the best but I don’t like the way that Visio 2013 makes it really difficult to add new members if you are making a class or an ER diagram. Visio 2013 is prettier, but a lot more clunky in my opinion. For Mac, I am a pretty big fan of Omnigraffle. It is simple, easy to use, has good stencils and gets the job done. For Linux, although there are various choices they all leave something to be desired. I recently tried out Visual Paradigm, even though it has a “free” version for non-commercial use, they put in gross watermarks if you make more than one of the same type of diagram. This is not very professional and I do not think that the software is good enough to pay for. It seems like they just took eclipse and added some drawing functionality. As much as I want to like Dia, it is just too clunky for every day use. I would not recommend this for anything other than very simple diagrams. yEd is a really neat tool, it is simple and free to use. This tool really stands out from the pack for me because it “just works”. It also gets extra points for using an open standard drawing format which makes it compatible with other standards based software. I think the best tool (but also the one with the highest learning curve) is Graphviz. Specifically, I am referring to using dot to make drawings. Despite the steep learning curve, it is 100% free software, standards based, is flexible and will draw exactly what you ask it to without too much trouble. Graphviz products embed perfectly in other programs which can be challenging when writing reports or papers. Also, if you master dot you will feel like a real hacker. If you make diagrams for work or school (UML, ER, etc) what tool do you use? Let me know in the comments below!

Convert Markdown to PDF in Sublime Text

2014-11-29 01:31:30 ][ Tags: hacking sublime text

Sublime Text is an awesome text editor that has a ton of useful extensions that can be installed through the Package Manager that make it even better. One of my favorite parts of emacs is org-mode, this allows you to organize your life, make awesome notes, and even perform spreadsheet calculations all from with in the emacs text editor. The best part about org-mode is that you can export your documents to HTML, PDF, and LaTex. In my opinion, sublime text is almost a modern successor to emacs because a lot of the extensions that have been developed for Sublime Text allow you to do fancy things like this as well. I am a huge fan of Markdown, It makes writing structured documents super simple, and is especially useful when your documents contain code snippets. As a CS student, I am often writing papers that include code snippets and this is not handled very well in most word processors. Org-mode does a great job of creating documents with code snippets, and I was excited to find out that with a few extensions to Sublime text you can make beautiful documents including code with little to no hassle. The following steps are being done with Sublime Text 3 running on Debian Testing, the steps may be a bit different for other Operating Systems. In order to make beautiful documents with code snippets in Sublime Text you will need to do the following.

  1. If you have not already install Package Control
  2. Install MarkdownEditing in Sublime Text
  3. Install Pandoc in Sublime Text
  4. Install Pandoc in Debian
  5. Install TexLive in Debian (this is to convert things to PDF)
sudo apt-get install pandoc texlive We are going to be leveraging the awesome tools that are provided by pandoc in order to make really cool things happen with our text editor! This even supports syntax highlighting! Once you have installed all of those pre requisites you can save a text file as markdown using the .md extension and then convert it to HTML, PDF, or other formats using pandoc. Simply open the command launcher with Control+Shift+P type in pandoc and select your output format. Now you can turn this: [caption id="attachment_125" align="alignnone" width="500"]Sample Python Code Sample Python Code[/caption] Into this: [caption id="attachment_124" align="alignnone" width="500"]Sample Python Code Pdf Sample Python Code Pdf[/caption] Awesome! Next steps are to level up your Sublime Text skills with Mastering Sublime Text.

Fake Web IDE with External Tools in Gedit

2014-06-08 00:18:45 ][ Tags: hacking

Gedit is my favorite text editor. I like that it is fast, reliable, cross platform, and has a ton of useful plugins and features. I am currently using it to work with LaTeX and unless I am working on a huge project, I will typically use gedit for all of my development work, specifically when it comes to web development. I use a lot of the plugins in gedit, but I have never used the External Tools plugin before. External tools is a very useful plugin because it allows you to do pretty much anything. The reason why I wanted to use it in the first place was to find a way to quickly launch HTML files that I was currently working on in a web browser, while also saving all of the changes to other HTML/CSS/JavaScript files that were related and currently opened in gedit. This is essentially what an IDE would do when you hit Run. In the past, I would just save all of my document and find the file in Nautilus to launch it. The problem with this approach is that I have an obsessive need to organize all of my projects into obscure and seemingly endless file paths. This can make it pretty difficult to find the file that I am looking for. The External Tools plugin solves all of these issues in a very elegant and simple way. So, without further ado, here is how you make a fake Web Development IDE in Gedit using External Tools.

  1. Enable External Tools: Edit –> Preferences –> Plugins
  2. Create a New External Tool: Tools -> Manage External Tools -> Hit the Plus sign
  3. Name the tool whatever you would like
  4. Assign it a shortcut key (optional)
  5. Set the following options on the bottom right
    • Save: All Documents
    • Input: Current Document
    • Output: None
    • Applicability: All Documents
  6. In the script editor, enter the following short script. This script will open the current document in your default web browser.
    #!/bin/sh x-www-browser $GEDIT_CURRENT_DOCUMENT_PATH 
  7. Close, and you are done!

You should now be able to use whichever shortcut you created to save all documents that you are working on, and open the current document in a web browser. This makes debugging much easier for web applications, and makes gedit a perfect lightweight web IDE. If you have some handy tips and tricks for gedit custom tools, please share in the comments below!