levlaz | лев | 列弗

As a part of my data science course on EdX we have been working with a lot of csv files. I spoke SQL long before I spoke Pandas and I find that it is much easier to do initial exploration of the data using raw SQL queries compared to the Pandas DSL.

Kaggle is a great repository full of useful data sets that are ripe for exploration. While a lot of these data sets come in both csv and sql flavors, some of them are CSV only. Using SQLit we are able to easily import these csv files into a database and then run queries for further data exploration. 

Im going to use the kickstarter data set for this tutorial, feel free to download the csv files from kaggle so that you can follow along. 

Pre Requisites

Make sure that you have SQLite installed before getting started with this tutorial. 

Steps to Convert CSV to SQLite

First, Download the data set from kaggle, this will come in the form of a zip file. Unzip this and open up a terminal in the directory where you have the new unzipped kickstarter-projects folder. 

In your terminal open up a new sqlite session followed by the name of the file that you want to save your new database to. 

sqlite3 ks.db

Inside of the sqlite shell, change the mode to csv. 

.mode csv

Import the csv file, and add the name of the table that you want the data to be imported into. 

.import kickstarter-projects/ks-projects-201801.csv ks

Verify that everything was imported correctly. Take a look at the schema, and first couple of rows. Your output should look something like this: 

sqlite> .schema ks

CREATE TABLE ks(
  "ID" TEXT,
  "name" TEXT,
  "category" TEXT,
  "main_category" TEXT,
  "currency" TEXT,
  "deadline" TEXT,
  "goal" TEXT,
  "launched" TEXT,
  "pledged" TEXT,
  "state" TEXT,
  "backers" TEXT,
  "country" TEXT,
  "usd pledged" TEXT,
  "usd_pledged_real" TEXT,
  "usd_goal_real" TEXT
);


sqlite> select * from ks limit 5;

1000002330|The Songs of Adelaide & Abullah|Poetry|Publishing|GBP|2015-10-09|1000.00|2015-08-11 12:12:28|0.00|failed|0|GB|0.00|0.00|1533.95
1000003930|Greeting From Earth: ZGAC Arts Capsule For ET|Narrative Film|Film & Video|USD|2017-11-01|30000.00|2017-09-02 04:43:57|2421.00|failed|15|US|100.00|2421.00|30000.00
1000004038|Where is Hank?|Narrative Film|Film & Video|USD|2013-02-26|45000.00|2013-01-12 00:20:50|220.00|failed|3|US|220.00|220.00|45000.00
1000007540|ToshiCapital Rekordz Needs Help to Complete Album|Music|Music|USD|2012-04-16|5000.00|2012-03-17 03:24:11|1.00|failed|1|US|1.00|1.00|5000.00
1000011046|Community Film Project: The Art of Neighborhood Filmmaking|Film & Video|Film & Video|USD|2015-08-29|19500.00|2015-07-04 08:35:03|1283.00|canceled|14|US|1283.00|1283.00|19500.00

Excellent! Now you can query this entire data set as your normally would. Happy data exploration! 

As a part of the visualization section of the python for data science course on EdX we watched this awesome video showing the health and wealth correlation of 200 countries over the last 200 years.

This is a fascinating look into the power of visualization, statistics, and data science. It is also a very interesting story that Rosling was able to convey in just 4 minutes.

Python for Data Science is an introductory course that provides an overview of various tooling that exists in the python world that is useful for data science purposes.

This includes things like:

• Jupyter Notebooks
• The numpy library
• The scipy library
• The pandas library
• The matplotlib library

The course provides an excellent overview to python, and suggests using anaconda which is a distribution of python geared toward data science purposes. Although this is a great way to get started with python if you have never used it before, installing multiple versions of python (which this approach would do) can be quite a pain to manage long term. This is especially true if you use python for other purposes such as web development with flask or django.

If you try to work through some of the jupyter notebooks that are presented in the course without installing anaconda, you will often see error messages like this:

---------------------------------------------------------------------------
ModuleNotFoundError Traceback (most recent call last)
in
1 get_ipython().run_line_magic('matplotlib', 'inline')
2 import numpy as np
----> 3 from scipy import misc
4 import matplotlib.pyplot as plt

ModuleNotFoundError: No module named 'scipy'

The solution is of course to install the package that is missing. In the example above we can install the missing package with pip3 install scipy.

You can use the one liner below to install all of the required packages for the course in one go. Note, this assumes you already have python3 and pip3 installed on your computer.

pip3 install jupyter pandas numpy scipy matplotlib imageio folium

I’m taking a data science course on EdX as a part of the UCSD MicroMaster program. I’ve always been curious in data science from a cursory point of view, but now I want to use it to analyze and understand business problems.

So far we have learned that data science is used in nearly every industry to provide actionable insights into complex problems. One of the most interesting things that I’ve seen so far is how data science is being used to help fight forest fires in real time.

Coming off a terrible week for wildfires where California faced the deadliest fire in its history, this application of data science seems more relevant and urgent than ever.