programming

Version 0.4.3 of Braindump, I Need More Functional Tests.

Yesterday, I shipped a new release of braindump. It was awesome. Until I tried to use it this morning and realized that notes were no longer being saved when you edit them. So today I cut a new release that fixes that bug, as well as implementing a new feature where you can see what notes you have shared in the past. This also addresses an issue with sharing notes, so you should be able to send notes via email again. Normally, when a product is being released on a regular cadence it is exciting. In this case, it just means that I really need more functional tests.

Post Mortem of Note Saving Bug

For those who are curious, I will document what happened in the hopes that it does not affect others in the future. Braindump was originally written using WTForms, which has CSRF protection built in so you don’t have to think about it. Recently, I have begun transitioning away from WTForms in lieu of regular forms that are submitted via AJAX. The most visible benefit is that you can see and edit all of your notes from the main page, instead of having to click into each note in order to edit it. After switching to AJAX, I quickly realized that I was no longer sending a CSRF token with POST and PUT requests, which is probably a bad thing. I enabled it in the previous release. Even though I got this to work in the login and registration modal (which are true forms), I didn’t consider the PUT request when editing a note. This is not a form at all, but a method that fires whenever you stop typing. Naturally, since there was no token and the endpoint was not marked as exempt, the response was a 400 error (missing CSRF token). My solution was to add the token as a meta tag to every page, and then send it along in the request headers of the AJAX call.

Enable CSRF

// app/__init__.py

csrf = CsrfProtect()

def create_app(config_name):
    ...
    csrf.init_app(app)
    ...

    return app

Token in the Header

// app/templates/app/app_base.html
// This works because csrf_token() is in the global scope.

<meta name="csrf-token" content="  csrf_token() " />

AJAX Method to add the token before sending the PUT request.

// frontend/js/src/braindump.editor.js

saveNote: function(id, content) {

        var csrftoken = $('meta[name=csrf-token]').attr('content')

        $.ajax({
                beforeSend: function(xhr) {
                        xhr.setRequestHeader("X-CSRFToken", csrftoken)
                },
                url: `/edit/${id}`,
                data: JSON.stringify({
                        'body': content,
                }),
                contentType: 'application/json',
                type: 'PUT',
                success: function (response) {
                        console.log(response);
                },
                error: function (error) {
                        console.error(error);
                }
        });
},

Everything now works fine, and in the future if I need to make more AJAX calls, the token will be available in the meta of each page. This would have been easily caught if I had had even a single functional test. I hope to have something in place over the next few days so that I stop shipping breaking changes. Braindump is still very beta level software. Once we make it past the 1.0 release, I will have close to 100% code coverage, full functional test suite, and develop a steady pace for incremental improvements so that there are no more surprises like this one in the future.

Standard
programming

Show all Flask Routes

One of my favorite things about rails is rake routes. This command will show you all of the routes that your application is currently aware of. I was searching for something similar in Flask and came across this snippet. Sadly, it did not work for me. I am using Python 3, so that may have something to do with it. In any case, I did not have time to dig into what was wrong, because just the other day I read this amazing introduction to functional python where my biggest take away was that as soon as you see multiple for loops, its time to use map. With my functional skills in tow, I wrote a simple one liner to print all of my routes.

@manager.command
def routes():
    import pprint
    pprint.pprint(list(map(lambda x: repr(x), app.url_map.iter_rules())))

Running this on braindump, shows me everything that I need to know.

[levlaz braindump]$ python3 manage.py routes
["<Rule '/auth/change-password' (GET, HEAD, OPTIONS, POST) -> "
'auth.change_password>',
"<Rule '/auth/change-email' (GET, HEAD, OPTIONS, POST) -> "
'auth.change_email_request>',
"<Rule '/auth/unconfirmed' (GET, HEAD, OPTIONS) -> auth.unconfirmed>",
"<Rule '/auth/register' (GET, HEAD, OPTIONS, POST) -> auth.register>",
"<Rule '/auth/confirm' (GET, HEAD, OPTIONS) -> auth.resend_confirmation>",
"<Rule '/auth/logout' (GET, HEAD, OPTIONS) -> auth.logout>",
"<Rule '/auth/login' (GET, HEAD, OPTIONS, POST) -> auth.login>",
"<Rule '/auth/reset' (GET, HEAD, OPTIONS, POST) -> "
'auth.password_reset_request>',
"<Rule '/empty-trash' (GET, HEAD, OPTIONS) -> main.empty_trash>",
"<Rule '/notebooks' (GET, HEAD, OPTIONS, POST) -> main.notebooks>",
"<Rule '/favorites' (GET, HEAD, OPTIONS) -> main.favorites>",
"<Rule '/settings' (GET, HEAD, OPTIONS) -> main.settings>",
"<Rule '/shutdown' (GET, HEAD, OPTIONS) -> main.server_shutdown>",
"<Rule '/archive' (GET, HEAD, OPTIONS) -> main.view_archive>",
"<Rule '/search' (GET, HEAD, OPTIONS) -> main.search>",
"<Rule '/trash' (GET, HEAD, OPTIONS) -> main.trash>",
"<Rule '/add' (GET, HEAD, OPTIONS, POST) -> main.add>",
"<Rule '/' (GET, HEAD, OPTIONS, POST) -> main.index>",
"<Rule '/static/bootstrap/<filename>' (GET, HEAD, OPTIONS) -> "
'bootstrap.static>',
"<Rule '/auth/change-email/<token>' (GET, HEAD, OPTIONS) -> "
'auth.change_email>',
"<Rule '/auth/confirm/<token>' (GET, HEAD, OPTIONS) -> auth.confirm>",
"<Rule '/auth/reset/<token>' (GET, HEAD, OPTIONS, POST) -> "
'auth.password_reset>',
"<Rule '/delete-forever/<id>' (GET, HEAD, OPTIONS, POST) -> "
'main.delete_forever>',
"<Rule '/notebook/<id>' (GET, HEAD, OPTIONS) -> main.notebook>",
"<Rule '/notebook/<id>' (DELETE, OPTIONS) -> main.delete_notebook>",
"<Rule '/favorite/<id>' (GET, HEAD, OPTIONS, POST) -> main.favorite>",
"<Rule '/restore/<id>' (GET, HEAD, OPTIONS, POST) -> main.restore>",
"<Rule '/archive/<id>' (GET, HEAD, OPTIONS) -> main.archive>",
"<Rule '/delete/<id>' (GET, HEAD, OPTIONS, POST) -> main.delete>",
"<Rule '/static/<filename>' (GET, HEAD, OPTIONS) -> static>",
"<Rule '/share/<id>' (GET, HEAD, OPTIONS, POST) -> main.share>",
"<Rule '/note/<id>' (GET, HEAD, OPTIONS) -> main.note>",
"<Rule '/edit/<id>' (PUT, OPTIONS) -> main.edit>",
"<Rule '/tag/<name>' (GET, HEAD, OPTIONS) -> main.tag>"]

Now, all that is left is for me to figure out why my API routes are missing. 🙂

Standard
programming

How to Ship a Simple Feature in Two Months

I started working on a feature to allow users to remove notebooks in Braindump two months ago. What started as adding an additional link and a corresponding flask endpoint to handle the deletion has turned into a horrible long running branch that I have no idea how to get out of. This is a common trap that I fall into, and I have seen other developers struggle with the same issues.

Scope

Add a Feature to allow users to remove notebooks.

Implementation

Easy Peasy, here is a link to delete notebooks. Ship it.

Side Tracking

Well, while we are here, maybe we should use the new “card” style that is hip now. Hmm, wait, this looks stupid in purple. Let’s make it blue. The cards look great, lets make the rest of the site blue as well. Ok it looks awesome in blue. Have not looked at CSS in a while, maybe we should refactor it, switch to SASS and concatenate all of the CSS into a single file. Oh crap, doing this breaks the outer theme compared to the inner theme since they were defining rules for the same elements. No problem, I will just rewrite the whole UI again.

30 commits later

What problem was I trying to solve again?

Conclusion

This is actually what happened. No matter how many times I tell myself I will never fall into this trap again, I keep on doing it. Next time will be different. In any case, the new notebook view looks good and allows you to delete notebooks. I hope to have this shipped this year.

Standard
programming

Find Dead Links on Your Jekyll Blog with HTML Proofer

Introduction

HTML Proofer is a super handy ruby tool that helps you check your statically generated HTML for any inconsistencies. If you have a large statically generated site then it is certainly worth setting this up because as your site continues to grow it will become more and more difficult to audit the validity of your pages. I have used HTML Proofer in the past, but for whatever reason I had “disable_external” set to true which ignored all outgoing links. It is still useful to find things like missing alt tags in images, and general invalid HTML, but this feature makes it a must for all blogs.

Configuration

Rather than clicking on every link on every page, let HTML Proofer do the heavy lifting for you with the following simple steps:

Install HTML Proofer

Add the following to your Gemfile

gem "html-proofer"
gem "rake"

Install all of your Gems

bundle install

Configure a Rake Task

Add the following to your Rakefile.

require 'html-proofer'

task :test do
  sh "bundle exec jekyll build"
  HTMLProofer.check_directory("./_site", {
    :allow_hash_href => true
    }).run
  end

Run the Task

bundle exec rake test

This will show you any failures and allow you to act upon them. Some sample output looks like this:

*  External link https://levlaz.org/tag/lxc/ failed: 404 No error
- ./_site/projects/index.html
*  External link http://ezbadge.levlaz.org/ failed: 301 Peer certificate cannot be authenticated with given CA certificates
- ./_site/salting-your-lxc-container-fleet/index.html
*  image /images/minions.jpg does not have an alt attribute (line 150)
- ./_site/setting-up-antlr4-on-windows/index.html
*  image /images/antlr.png does not have an alt attribute (line 156)
*  image /images/grun.png does not have an alt attribute (line 163)
- ./_site/share-this-on-facebook/index.html

Configure CI

If are using CircleCI you can add the following to your circle.yml to run the proofer automatically.

test:
  override:
    - bundle exec rake test

The proofer returns an exit code of 1 upon failure, so this is a great way to enforce quality before deploying your site to production.

Conclusion

There is nothing worse than clicking on a link and seeing a 404. When the post is from 2013, perhaps you can excuse it, but it is still a terrible experience for the user and as a “web master” you owe it to your users to prevent link rot.

Standard
writing

Portable MFA

A couple of years ago I got the portable MFA book for Kindle. I was really into writing back then. I kept a blog (which I have since completely deleted) called “The Thoughts Bubble” where I would write poetry, prose, short stories, and thoughts about writing in general. I think I even wanted to just become a writer at some point.

My Kindle has been sitting on my bookshelf, dead, for several months now. This does not mean I have not been reading, I have just not been reading on my Kindle. I have been fluctuating between Apple iBooks, Physical Books, and random free ebooks that I find online. I finally charged up my Kindle last night and started to read the first of the Foundation series by Isaac Asimov. As I was looking through my Kindle Library, I stumbled across this book again and decided to read it again.

I miss being younger, and having ideas. When I read this book the first time I was able to really take advantage of many of the exercises and fondly remember exercising my writing skills. Now, I am stuck on the very first exercise called Poem, Dream, Conflict. Essentially you think of a quote, a recent dream, and a recent conflict that you have had and write a couple unrelated paragraphs of prose. Then you put them all together without explicitly referencing anything. The main idea here is that when it comes to writing short stories, the more questions you leave open to the reader the more compelling your work becomes.

I don’t remember the last time I read a poem. All of my dreams are horrible but I cannot recall any specific reason why. I try to avoid conflict. I spent a good hour staring at a blinking cursor while trying to complete this exercise. I gave up and started reading some poetry instead.

Standard
programming

Coursera Rails Module 2 Notes

Intro to Ruby

This module goes through a general introduction to Ruby. One thing that I find interesting is how optimistic Ruby is, in the sense that everything except for false and nil is true. This is especially interesting from the perspective of comparing true and false in sqlite where sqlite stores 0 as false and 1 as true. I have not had to deal with this yet, but it is nice to keep in mind going forward. Another nice thing that I have not used that much is irb this is basically a Ruby repl that allows you to quickly test out concepts in your shell. I also love (since I work at a CI company) that the very first lesson about ruby talks about rspec and even has us use it go grade our assignments. It is never too early to start thinking about tests.

Tour of Ruby Data Types

The next module took a nice tour of Ruby data types. I learned some pretty neat things.

  • Strings in Ruby super expressive, string interpolation is pretty amazing and very clean. Destructive Methods Have !, I like this convention, you see it a lot in Clojure.
  • %w can make multiple strings into an array (like using .split() in JS)
  • You can get a random element from an array with .sample, pretty neat
  • Hash Order is maintained, this is really powerful

I also finally understand how blocks in Ruby (closures) work. (After many years of derping through Vagrantfiles)

Object Oriented Ruby

  • All methods public by default
  • Clean class creation with useful helper methods for getter/setter
  • Talked about how most of the functionality in Collections comes from using the Enumerator mix-in, gave an example of how to use it, super powerful!

Unit Testing

Again, I love that this course started right off the bat talking about unit testing. A+ to the course creators for covering this topic. The instructor did a good job talking about the differences between Test::Unit, MiniTest, and RSpec with great examples of how to use all of them.

Assignment

The final assignment for this module was to write a program that reads a file and calculates word frequency. Overall it was pretty simple, I think the material did a good job preparing us for the task, and caused us to use a lot of the different tools that we learned about Ruby.

Parting Thoughts

I have often heard that Ruby was designed for developer happiness. I really love the clean syntax, well documented API, and elegance. The next module will be a deep dive into Rails, looking forward to using more of the stuff that I have learned in this Ruby crash course.

Standard
programming

Coursera Rails Module 3 Notes

I just finished the final module of this course. Overall I think it was a great course that provided a good overview of Ruby, and a thorough introduction to Rails. The next course dives into working with Databases and I am looking forward to learning more about Active Record.

Some Interesting Facts

  • Rails has only been around since 2004.
  • Startups love Rails since it makes it easy to do rapid prototyping.
  • MVC has been around since 1979

Useful Resources

The lectures had a bunch of useful tools sprinkled throughout.

Things I love about Rails

  • Convention over Configuration. I love this, I love best practices, and I love having some guidelines to follow. Think less, do more.
  • ORM makes it easy to swap out your DB as times goes on. Start dev in SQLite and should be able to seamlessly transition to MySQL, PostgreSQL, etc.
  • Rake = Ruby Make, comes with a ton of useful built in utilities. For instance if you want to see all of the routes that your application is aware of you can run rake routes.
  • Helpers (or view helpers) are reusable macros that can be used across any of your views. This is much cleaner approach than themacros that I am used to working with in Flask. There are at on of useful helpers built into rails, like “cycle” which makes it super easy to make striped HTML tables.

Things I learned

General Rails Tooling

  • Finally have a solid understanding of how Gems, Bundler, and Rbenv work together.
  • Learned about and used the HTTParty Gem
  • You can use a Pessimistic Version Constraint (Love the name) in your Gemfile which basically says do not go over a certain version. Essentially you are betting against the house and saying that you believe that an update to one of your gems is going to break everything sometime in the future. (Trust me this happens all of the time).

Testing with Rspec and Capybara

Once again, I cannot say how happy I am that this course dives right into all sorts of testing. Kudos to the JHU team for thinking about this important skill even in an introductory course. This module talked about doing acceptance testing with Capybara. We learned about:

Debugging

The built in debugging tools in development mode in rails are just awesome. You get an IRB console inside of the browser, this is super useful for debugging purposes. The gems that are responsible for this functionality are byebug and web-console.

Deploying To Heroku

The course walked us through how to deploy an app to Heroku. Super simple of course. It is really nice to be able to see your application out in the wild!

Pro Tip

The rails generator makes it super easy to create all of the files that you may need for a controller or model. However if you accidentally create a model with the wrong name and your project it huge it can be somewhat daunting to figure out what is safe to remove. You can reverse a rails generated command with:

rails d $TYPE $NAME

So for example, if I created a model called recipeee

rails g model recipeee

I could remove it with

rails d model recipeee

I can see how this will come in handy in the future.

Final Project

The final project is a simple web application called Recipe Finder that allows you to search for recipes using the food2fork api. I cleaned it up a bit and added an actual search bar (the assignment just had you add query params).

Standard
software

Notes on Steve Krug’s ‘Don’t Make Me Think’

I am working my way through Jeff Atwoods reading list and finally got around to reading Don’t Make Me Think{.add-link} by Steve Krug. This is one of the best books that I have read in quite some time. His writing style is amazing, and there is so much useful content in the book that I don’t even want to spoil it by talking about it here. As a completely unrelated side note, Steve Krug is a Usability Professional, the last time I read about this specific term was in Cory Doctorow’s Eastern Standard Tribe where the main character had the same career.

Hot Takes

  1. People don’t read your website, stop designing it like they do. Instead, learn how to make great billboards.
  2. Focus on the problem you are solving rather than the problem that has already been solved.

There is a common trope these days on Twitter where people joke about the fact that every product website looks the same. Jumbotron header, green call to action, and a three column list with icons that explain what your product does. For instance: Mockup of Every Website Ever
Made I see this now as a less of a problem, this model works, its clear, use it and focus on building your product instead of re-inveting the wheel of what product pages are supposed to look like.

  1. ~~Write Less. My favorite part of this book is the heading to Chapter 5; Omit Needless Words. I learned about this when I read On Writing Well{.add-link} during an expository research writing course in college, but based on how much text is in this paragraph, I seem to have forgotten the lessons that the book taught me~~.
  2. Write Less.
  3. Don’t punish users for not doing things your way. This is especially true for forms of all types.

Don’t make me jump through hoops just because you don’t want to write a bit of code.

  1. Think about Accessibility. I am guilty of this myself, but primarily because all of my projects are half finished. The main things to watch out for are color coded things, inability to adjust text size, and making things impossible to read via a screen reader. (Like by using HTML tables for structure). Ricardois famous for thinking about these types of things in his projects and it motivates me to do better.

Further Reading

I am one of those strange people that read the bibliography, further reading section, and hang on to each recommendation. This is partially why my current reading list is too long to even look at. With that being said, a couple books that I plan to read per Krug’s recommendation are:

  1. The Design of Everyday Things{.add-link} — Classic Book about design. If you look at some of my work (this blog, braindump) you will immediately realize that I am not a designer. I am OK with that, I hope this book will help me get a better understanding of the fundamentals.

Update: By the time I finished this post, I already bought the book at my local bookstore. Can’t wait to read it.

  1. Letting Go of the Words{.add-link} — how to become a better writer for the web.
  2. Forms that Work: Designing Web Forms for Usability{.add-link} — I have not heard about this book before, but as someone who has used and designed a bunch of awful forms this seems like a great use of my time.
  3. Rocket Surgery Made Easy{.add-link} — Don’t Make Me Think had a great overview of DIY usability testing, this is a follow up from Krug that dives deeper into Usability Testing for teams that cannot afford the \$5-10K cost of a professional usability test.
  4. A Web for Everyone: Designing Accessible User Experiences{.add-link} — The chapter about Accessibility in Don’t Make Me Think really hit home for me. I look forward to learning more about the accessibility problems that I do not think about that often and apply them to my own work.
Standard
software

Notes on Mastering Sublime Text

I picked up Mastering Sublime Text while working through the first section of the Rails Series on Coursera. First, its nice to sometimes switch up your text editors to see what you are missing. Second, Sublime Text has a very small learning curve and I never took the time to learn about some of the more advanced features so this was a decent use of my time. Third, I really want to write a CircleCI plugin for Sublime Text similar to the ones that exist for atom and Visual Studio Code, and this book has a great section that goes into detail about what it takes to write a plugin.

Things that I always loved about Sublime Text

  1. Multi-line Selection
  2. Dead Simple Multi File Find and Replace
  3. Speed — Go try to open a 10K line log file in Atom. I’ll wait.
  4. Package Control package manager was one of the first of its class for GUI editors. Go try to install any plugin in Eclipse. I’ll wait.
  5. Distraction Free Mode — Perfect for writing prose

Things that I wish Sublime Text had after using Atom

  1. Color coded indication about git status. There seems to be afeature request for this, but I am not 100% sure of the reasoning on why this is so challenging to implement.
  2. File Type Icons; I don’t know why but this is one of my favorite atom plugins and it makes quickly finding the files that you are looking for a breeze.
  3. Pigments plugin – another one of my favorites. ST has a similar plugin but its not as good and it also underlines the font which can be a big hard to read.
  4. Being able to drag and drop files around in the sidebar. This comes in handy quite a bit.
  5. Better overall git support, I actually think that VS Code has some of the best git integration out there, followed by magit and vim fugitive.

Things I learned

Searching

  1. Sublime text has incremental search. This was one of my favorite features of emacs. Command + I opens it up, and you can go to the next result by repeating Command + I.
  2. When searching for stuff, if you double click on the result it takes you directly to the file. This is such an awesome feature.
  3. Go to anything with Command + P, holy crap is this powerful.

Editing

  1. This did not come directly from the book, but I learned this cool trick while working on this post. If you highlight some words you can then wrap them in braces. This is super handy when adding links in markdown.
  2. Snippets; I have triggered snippets accidentally in the past, but learning more about how they work and how to make custom ones was very useful.
  3. Vintage mode makes sublime behave like vi. Pretty cool!

Plugins

  1. SublimeCodeIntel – one of the best features of VS Code isIntelliSense, this comes close and is a super powerful code-completion plugin.
  2. TrailingSpaces – Linus Torvalds tells developers who use any editor that does not automatically strip trailing white space to get a real editor. This is built into Sublime Text, but this plugin adds some additional features to make this even better.

Sublime Text Internals

  1. Sublime has a bunch of useful Environment Variables that can be used when developing snippets and plugins.
  2. Visual Cues, I mentioned earlier that I really like some of the visual cues that Atom offers. There are a couple built in settings like making folders bold and highlighting modified tabs that make things easier to look at.

Testing

  1. The biggest difference between mode IDE’s and Text Editors is build in testing and building functionality. Although testing is not built into Sublime Text, the book did an excellent job talking about several plugins that make testing PHP, Python, and Ruby a breeze. I found the RubyTest package to be the most useful since it has built in support for Test::Unit, Rspec, and Cucumber.
  2. There are also a ton of great plugins for debugging from within Sublime Text.

Plugin Development

  1. This was my primary reason for getting this book since I want to write a CircleCI Plugin that shows the status of your most recent build in the window. I was not disappointed, the book did a great job discussing what it takes to start, develop, and publish a plugin.

Errata

  1. SublimeLinter; the book mentions this plugin and says that it has a bunch of linters build in, this is no longer the case. It’s still a very useful tool, it just requires a bit of configuration to get the most value out of it.
  2. The book tells you to add trim_trailing_white_space_on_save to your user settings if you have installed the TrailingSpaces packages. This did not work for me on Sublime Text 3. Instead I needed to add this setting under the trailing_spaces.sublime-settings file with the new key called"trailing_spaces_trim_on_save"

Parting Thoughts

Mastering Sublime Text{.add-link} is a quick read, full of useful information for new and experienced users of Sublime Text. I would highly recommend it for anyone who uses Sublime to serve as a deep dive introduction and as a handy reference manual.

Standard
programming

Become a Rails Developer

I am working through this Rails Series on Coursera to get a more structured exposure to the rails ecosystem. I find that I learn best when I have a school like structure, so even though I have been fiddling around with Rails for quite some time now I am hoping that this series will teach me some good patterns for the future.

So far I have been super impressed. First, I had no idea that Johns Hopkins even had a Computer Science department. Second, watching the introductory video on this series made me envy the students in that program. I finished my MS in Computer Science at NOVA Southeastern University last year and although I would highly recommend this program for anyone who wants to learn the intricacies of CS, it is not the best program to prepare you for a programming job. This is not that programs fault but instead a general problem with CS education. Most programs are about 10 years behind in terms of trends, tooling, and practices. However, based on the course intro which was given by the faculty of JHU I feel like they “get it”. I am sure that the JHU program is not that much different as far as core content, but the fact that they give their students exposure to real world practices (rails, mongo, git, etc) is inspiring.

I am looking forward to working through this course and becoming a jr rails developer. I hope to apply these skills directly to this projectthat I started two months ago and have not touched since. Opendesk is an ambitious project to make a support center that does not rely on tags to accomplish anything outside of the “norm”. Hopefully be the end of this series I will have enough knowledge and skills to push this project over the edge and make an actual release.

One immediate benefit of this course is that I learned that Pro Git is actually available as a free ebook. This is an awesome resource that really digs into the intricacies of git. I would consider myself an intermediate git user, but there is always room to learn more.

Standard