Sofia Heisler No More Sad Pandas 

In this excellent talk, Sofia Heisler explains how to optimize Pandas code. There is a striking difference between a naive implementation and an optimized one, a nearly 500x improvement.

She also described two very useful benchmarking functions, timeit, and line_profiler.

I wonder how much of this would apply to “normal” python development? If nothing else, the two benchmarking functions will certainly come in handy in future optimization.

Posted in data science | Tagged | Leave a comment

Six Foot Long Scorpions

There was a super interesting article in the SF Chronicle today about scorpions. It profiled Lauren Esposito, an arachnologist at the California Academy of Sciences, and her work studying these creatures. The venom found in some species of scorpions has been used to provide us with insights in fighting disease and developing opioid free pain relief. She discussed how original scorpions were over six feet long. Can you image?

It all began, she said, about 450 million years ago, when the ancestors of scorpions became the first arthropods to leave the sea. At the time, they were monsters, up to 6½ feet long, with stingers to match.

Source: Bay Area scientist stung by public’s perception of scorpions as scary scourge – SFChronicle.com

 

Posted in science | Leave a comment

Installing Python Packages for DSE200x with pip

Python for Data Science is an introductory course that provides an overview of various tooling that exists in the python world that is useful for data science purposes.

This includes things like:

  • Jupyter Notebooks
  • The numpy library
  • The scipy library
  • The pandas library
  • The matplotlib library

The course provides an excellent overview to python, and suggests using anaconda which is a distribution of python geared toward data science purposes. Although this is a great way to get started with python if you have never used it before, installing multiple versions of python (which this approach would do) can be quite a pain to manage long term. This is especially true if you use python for other purposes such as web development with flask or django.

If you try to work through some of the jupyter notebooks that are presented in the course without installing anaconda, you will often see error messages like this:

---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
in
1 get_ipython().run_line_magic('matplotlib', 'inline')
2 import numpy as np
----> 3 from scipy import misc
4 import matplotlib.pyplot as plt

ModuleNotFoundError: No module named 'scipy'

The solution is of course to install the package that is missing. In the example above we can install the missing package with pip3 install scipy.

You can use the one liner below to install all of the required packages for the course in one go. Note, this assumes you already have python3 and pip3 installed on your computer.

pip3 install jupyter pandas numpy scipy matplotlib imageio folium
Posted in data science | Leave a comment

Data Science Helps Fight Wildfires

I’m taking a data science course on EdX as a part of the UCSD MicroMaster program. I’ve always been curious in data science from a cursory point of view, but now I want to use it to analyze and understand business problems.

So far we have learned that data science is used in nearly every industry to provide actionable insights into complex problems. One of the most interesting things that I’ve seen so far is how data science is being used to help fight forest fires in real time.

Coming off a terrible week for wildfires where California faced the deadliest fire in its history, this application of data science seems more relevant and urgent than ever.

Posted in data science | Leave a comment

Hot Dog Coma

I’ve been in New York City for two days now. One of the greatest cities in the world; center of the universe, a culinary heaven, full of Michelin star restaurants, and all I keep eating is Gray’s Papaya Hot Dogs and cheap slices of pizza.

I’ve been walking in circles around the 4-5 block radius between 35th and 45th street and 8th avenue with papaya juice in one sauerkraut smelling hand and a cigarette in the other.

There is a mustard stain on my hoody. I am getting ready to fall into a hot dog coma. I couldn’t be happier.

Posted in travel | Leave a comment

Continental Freshwater People

In a recent blog post on the Vonnegut Library blog, Rai Peterson, an English Professor at Ball State University writes about the perception of how writers only come from

“that cluster of states on the east coast that were so small their names on maps had to be written on the ocean.”

One of my favorite quotes from “A Man Without a Country”, is when Vonnegut describes himself as a “continental freshwater person”. This is one of the more interesting ways that I have heard someone describe the mindset that comes with growing up in the midwest. Despite my love for Vonnegut’s work, I am a bit ashamed to admit that until I visited the Vonnegut Museum and Library I had no idea that he was from Indianapolis.

Growing up in Cincinnati Ohio, I also dreamed of becoming a writer and the sentiment that Peterson describes rings true to me to this day.

Later in life, like many others, I escaped the continental freshwater, and found my way to a city by the ocean. Over the last few years I have been visiting every state capital and reading a ton of books along the way. Vonnegut, along with all of the other writers in those lists who did not come from a city by the ocean give me a lot of hope that although writers may end up in the same place, but they can come from anywhere.

Posted in writing | Leave a comment

Lies

I saw a man this morning burning a bible on the sidewalk. He had a blanket over his head and was screaming “LIES” into the street.

Commuters passed him by on their way to work like nothing interesting was going on. Just another day in San Francisco.

I wonder which translation it was.

Posted in thoughts | Leave a comment

Killed the bird

I deleted Twitter the other day. I got tired of the bullshit.

I wanted to tell the world that I did this. But I have no idea how to do that now that I don’t have Twitter.

I’ll probably post brief thoughts here more often. Or better yet, I’ll keep them to myself.

Posted in thoughts | 2 Comments

Bukowski On Writing

I finished reading a book by Charles Bukowski which was supposed to be about his thoughts on writing. What it turned out to be was a collection of stories, letters, and book forwards over the years that are tangentially related to writing.

I enjoyed the book overall but I think that only a Bukowski fan would really appreciate it.

My biggest takeaway from the book is that if I want to be more like Bukowski then I should listen to classical music and drink more wine.

Posted in books, writing | Tagged | Leave a comment

Docker Based Development Environment for Packaging

I started a new repo called sid-builder which provides a docker based development environment for debian packaging. This project follows all of the best practices that were outlined in the Guide for New Maintainers. The benefit of this approach is that you should be able to easily reproduce your development environment on any machine that supports docker. It also allows you to easily run sid without worrying about breaking your system.

I have not actually built any packages using this setup just yet, but I am going to give it a try over the next few days and report back here.

Posted in debian | Leave a comment