Programming, Technical, Uncategorized

Installing scikit-learn; Python Data Mining Library

Update: The instructions of this post are for Python 2.7. If you are using Python 3, the process is simplified. The instructions are here:

Starting with a Python 3.6 environment.

Assumptions (What I expect to already be installed):

  1. Install numpy: pip install numpy
  2. Install scipy: pip install scipy
  3. Install sklearn: pip install sklearn

Test installation by opening a python interpreter and importing sklearn:
python
import sklearn

If it successfully imports (no errors), then sklearn is installed correctly.

Introduction

Scikit-learn is a great data mining library for Python. It provides a powerful array of tools to classify, cluster, reduce, select, and so much more. I first encountered scikit-learn when I was developing prototypes for my first business venture. I wanted to use something that was easy and powerful. Scikit-learn was just that tool.

The only problem with scikit-learn is that it builds off of some powerful-yet-finicky libraries, and you will need to install those libraries, NumPy and SciPy, before you can proceed with installing scikit-learn.

To a novice, this can be a frustrating task since the order of installation matters and many Google searches will only produce unhelpful and long-winded responses. Thus, my motivation to set the record straight and provide a quick tutorial on how to install scikit-learn — mostly on Windows, but I have provided links and notes on both Linux and Mac installations as well.

In the process of this tutorial, you will install (or already have) the following — in this order:

NOTE: I have provided the links unlabeled above because, like all tech/installation tutorials, over time they become obsolete. By providing the links as they are, it is my hope that even if new versions come out, you will be able to use this tutorial to find the resources you need.

Step 1: Install Python

If you do not already have Python, install it now at the address provied above (https://www.python.org/downloads/). I will be using Python 2.7 for this tutorial.

The installer for python is quick and good. Once installed, we will need to check to see if Python is available on the command line. Open a terminal by searching for ‘cmd’ or running C:\Windows\System32\cmd.exe. I would recommend creating a shortcut if you are doing this a lot.

in the command line, enter:

python –version

something similar to “Python 2.7.6” should display. That shows that python is working and accessible from the cmd line.

Step 2: Install NumPy

NumPy is a powerful library for Python that contains advanced numerical capabilities.

Install NumPy by downloading the correct installer using the link provided above (http://sourceforge.net/projects/numpy/files/NumPy/1.10.2/) then run the installer.

NOTE: There are a few installers based on your OS version AND the version of Python you have. It is important that you find the right installer for your OS and Python version!

Step 3: Install SciPy

Download the SciPy installer using the link provided above (http://sourceforge.net/projects/scipy/files/scipy/0.16.1/) and run it.

NOTE: There are a few installers based on your OS version AND the version of Python you have. It is important that you find the right installer for your OS and Python version!

Step 4: Install Pip

Pip is a package manager specifically for Python. It comes in handy so much that I highly recommend that you install it to help manage python packages.

Go to the link provided above (https://pip.pypa.io/en/stable/installing/).

The easiest way to install pip on Windows is by using the ‘get_pip.py’ script and then running it in your command line:

python get_pip.py

If you are on Linux you can use apt-get (or whatever package manager you have):

sudo apt-get install python-pip

Step 5: Install scikit-learn

NOTE: More information on installing scikit-learn at the link provided above (http://scikit-learn.org/stable/install.html)

On Windows: use pip to install scikit-learn:

pip install scikit-learn

On Linux: Use the package manager or follow the build instructions at http://www.bogotobogo.com/python/scikit-learn/scikit-learn_install.php

Step 6: Test Installation

Now we must see if everything installed correctly. Open up a command line terminal and type:

python

This will open a python interpreter. You will know this because there will be some text and three chevrons, “>>>”, prompting input. Type:

import sklearn

If nothing happens and another prompt appears scikit-learn has been installed correctly.

If an error occurs, there might have been a mis-step in the process. Go back through the tutorial to see if any steps were missed or follow the error message that was given.

Standard
Programming, Technical

Git: List all branches, who made them, and when they were created

I ran into an interesting problem today that took a bit longer to figure out, and so I will post it here in the hopes that others will find the solution faster than I did.

My problem: I am inspecting a rather large and complex piece of software and need to look at its history – more specifically, I need to figure out when certain branches were created and who created them.

Now, I could have just hacked together a throw-away solution, but that’s not who I am. As I was looking, I found that there didn’t seem to be much on this – aside from the Stack Overflow questions.

So I whipped up a little script that gets all of the branches in the git repo and lists the details of their first commit. These details include the date, and creator – which is what I needed. (Bash script)

#
# Author: Caleb Shortt
#
# For some reason, lists the files and directories'
# history as well...(They're not all branches)
#
# Sources:
# Stack Overflow: Question: "What is the easiest/fastest way to find out when
# a git branch was created"
#

readarray -t branch_list <<< $(git branch -a)

for branch in ${branch_list[@]}
do
    echo '---------------------------------------------'
    echo "Branch: $branch"
    echo '---------------------------------------------'
    git show --summary `git merge-base master $branch`
done

Now there is a little hiccup with the script also printing out the details of whatever is in the directory, but the details for the branches are in there. I’ll update this if I figure out why it’s doing that.

Standard
Programming, Technical

Auto-generate HTML using a tree in Python

Here’s something I’ve been working on and thought was interesting.

I needed to dynamically generate HTML with varying degrees of nesting and attributes in Python. All I found was a few Stack Overflow questions on generating HTML – and some blogs talking about hard-coding the tags.

This is far from what I needed, so I started to look into making my own – besides, it looked fun!

I started by looking at what I actually wanted:

  • Needs to be easily extensible
  • Needs to be structured in a way that makes sense

With these requirements in mind, I elected to go the tree approach. HTML works much like a hierarchy: since the <html> tag is the root node and the <head> and <body> nodes are children of <html> and so on. Obviously it would be an n-ary tree as a node can have lots of children – or none.

Also, I would have to set up a specific tree traversal that would execute commands pre and post traversal – Namely printing out the beginning and ending tags.

This is what I came up with for the tree structure:


class DefTreeNode(object):
    '''
        A Protocol Definition Tree Node.
        It will contain a single payload (ex: a tag in HTML),
        a list of child nodes, a label, and what to do prefix
        and postfix during traversal.

        The payload is of type 'Generic_HTML_Tag'

        The label is a unique string that identifies the node.
        Both the label and the payload are required to initialize
        a node.
    '''

    def __init__(self, label, payload, contents=""):
        self.children = []
        self.payload = payload
        self.label = label
        self.contents = contents

    def addChild(self, child):
        if child:
            self.children.append(child)
            return True
        return False

    def setPayload(self, payload):
        if payload:
            self.payload = payload
            return True
        return False

    def setLabel(self, label):
        self.label = label

    def setContents(self, contents):
        self.contents = contents

    def getChildren(self):
        return self.children

    def getPayload(self):
        return self.payload

    def getLabel(self):
        return self.label

    def getPrefix(self):
        if self.payload:
            return self.payload.getPrefix()
        else:
            return None

    def getPostfix(self):
        if self.payload:
            return self.payload.getPostfix()
        else:
            return None

    def getContents(self):
        return self.contents

Note that the payload is not a string but (as the class descriptor comment says) is of type ‘Generic_HTML_Tag’. We will get to that in a second. Let’s finish the tree structure first.

Now that I have a node to work with, let’s make the tree. The tree class will contain the traversal code and a “find node” function, and it will hold the root node for the structure:


class DefinitionTree(object):
    def __init__(self, node):
        self.root = node

    def getRoot(self):
        return self.root

    def findNode(self, label):
        return self.recursive_findNode(label, self.root)

    def traverse(self):
        if self.root:
            return self.recursive_traverse(self.root)
        else:
            return ""

    def recursive_traverse(self, node, construction=""):
        '''
            Traverse the tree.
            This algorithm will run a pre-order traversal. When
            the algorithm encounters a new node it immediately
            calls its 'prefix' function, it then appends the
            node's payload, and traverses the node's children.
            After visiting the node's children, its 'postfix'
            function is called and the traversal for this node
            it complete.

            Returns a complete construction of the nodes' prefix,
            content, and postfix in a pre-order traversal.
        '''

        if(node):
            construction = construction + node.getPrefix() + node.getContents()

            for child in node.getChildren():
                construction = self.recursive_traverse(child, construction)

            return construction + node.getPostfix()

    def recursive_findNode(self, label, node):
    '''
        Executes a search of the tree to find the node
        with the specified label. This algorithm finds the first
        label that matches the search and is case insensitive.

        Returns the node with the specified label or None.
    '''

        if(node is None or node.getLabel().lower() == label.lower()):
            return node

        for child in node.getChildren():
            node = self.recursive_findNode(label, child)
            if node is not None:
                return node


Now, let's create the payload for the nodes. This will be done by creating a file to store our 'html tag classes':

 


class Generic_HTML_Tag(object):
    '''
        A Generic HTML tag class
        This class should not be called directly, but contains 
        the information needed to create HTML tag subclasses
    '''

    def __init__(self):
        self.prefix = ""
        self.postfix = ""
        self.indent = 0
        self.TAB = " "

    def getPrefix(self):
        return self.prefix

    def getPostfix(self):
        return self.postfix

    def setPrefix(self, prefix):
        self.prefix = prefix

    def setPostFix(self, postfix):
        self.postfix = postfix


class HTML_tag(Generic_HTML_Tag):
    def __init__(self, indent_level=0):
        Generic_HTML_Tag.__init__(self)
        self.indent = indent_level
        self.generateHTMLPrefix()
        self.generateHTMLPostfix()

    def generateHTMLPrefix(self):
        self.prefix = self.indent*self.TAB + "<html>"

    def generateHTMLPostfix(self):
        self.postfix = self.indent*self.TAB + "</html>"

class HEAD_tag(Generic_HTML_Tag):
    def __init__(self, indent_level=0):
        Generic_HTML_Tag.__init__(self)
        self.indent = indent_level
        self.generateHTMLPrefix()
        self.generateHTMLPostfix()

    def generateHTMLPrefix(self):
        self.prefix = self.indent*self.TAB + "<head>"

    def generateHTMLPostfix(self):
        self.postfix = self.indent*self.TAB + "</head>"

class TITLE_tag(Generic_HTML_Tag):
    def __init__(self, indent_level=0):
        Generic_HTML_Tag.__init__(self)
        self.indent = indent_level
        self.generateHTMLPrefix()
        self.generateHTMLPostfix()

    def generateHTMLPrefix(self):
        self.prefix = self.indent*self.TAB + "<title>"

    def generateHTMLPostfix(self):
        self.postfix = self.indent*self.TAB + "</title>"

class BODY_tag(Generic_HTML_Tag):
    def __init__(self, indent_level=0):
        Generic_HTML_Tag.__init__(self)
        self.indent = indent_level
        self.generateHTMLPrefix()
        self.generateHTMLPostfix()

    def generateHTMLPrefix(self):
        self.prefix = self.indent*self.TAB + "<body>"

    def generateHTMLPostfix(self):
        self.postfix = self.indent*self.TAB + "</body>"

class P_tag(Generic_HTML_Tag):
    def __init__(self, indent_level=0):
        Generic_HTML_Tag.__init__(self)
        self.indent = indent_level
        self.generateHTMLPrefix()
        self.generateHTMLPostfix()

    def generateHTMLPrefix(self):
        self.prefix = self.indent*self.TAB + "<p>"

    def generateHTMLPostfix(self):
        self.postfix = self.indent*self.TAB + "</p>"

 

Great! Obviously this is quite basic and incomplete, but the idea is there. The ‘generateHTMLPrefix() / Postfix() functions are the modifiers. You can add extra parameters and such here without crashing your code elsewhere. Also, you can add additional logic to, say, only add a specific attribute if the tag has it.

Ex: only a few tags have an onload attribute

 

Now that you have, let’s say, a file with your tree and node called HTMLTree.py and a file with your html markup classes called HTML_Tags.py, let’s bring it all together:

import HTML_Tags
import HTMLTree


if __name__ == "__main__":

    html = DefTreeNode("html_tag", HTML_Tags.HTML_tag())
    head = DefTreeNode("head_tag", HTML_Tags.HEAD_tag())

    head.addChild(DefTreeNode("title_tag", HTML_Tags.TITLE_tag(), "basic title here!"))
    html.addChild(head)

    body = DefTreeNode("body_tag", HTML_Tags.BODY_tag())
    body.addChild(DefTreeNode("p_tag", HTML_Tags.P_tag(), "paragraph here"))
    html.addChild(body)

    htmltree = DefinitionTree(html)

    searchlabel = "p_tag"
    print 'Searching for a node labeled: ' + searchlabel

    node = htmltree.findNode(searchlabel)
    if(node):
        print "\n\n" + node.getPrefix() + node.getContents() + node.getPostfix()
    else:
        print '\n\nNode with label \"' + searchlabel + '\" not found!'

    print '\n\nTree Traversal:\n'

    print htmltree.traverse()

The output should look similar to this:

Searching for a node labeled: p_tag


<p>paragraph here</p>


Tree Traversal:

<html><head><title>basic title here!s_static</title></head><body><p>paragraph here</p></body></html>

Great! You dynamically generated HTML in an extensible way!

Standard
startups, Technical

Startup Series: Episode 1

Surprise!

The distant memory that is my last post is not due to me forgetting about this beautiful little bog. I am in the process of starting a new company and, between that and my thesis work, I have not had much time to work on new posts. But alas I am back  – and with an exciting new series that chronicles my journey through the thrilling roller coaster that is the start-up experience!

I have some planned episodes (including this one) and have reserved room for additional episodes as time, and challenges, go by. This series will address issues, main concepts, and directions that I have encountered while running my start-up. Also, for those who are interested, my new company is called R-Gauge Metrics Inc.

Along the way I will try provide some reading material that has helped me with my company. Maybe they will help with yours also.

All-righty then! Lets get started!

 

Episode 1: Inception

Inception: The establishment or starting point of something; the beginning.

I, and I’m sure you have too, have met some people while cruising through life who were extra-keen on starting their own company. This is usually exciting for me and I love to see such motivated individuals strive towards their goals. However, upon further inquiry, I sometimes notice that the person just wants to start their own company for the sake of starting their own company. Their drive is strong and their intentions are good, but without a clear problem to solve, or product to sell, they more resemble the captain of a ship that leaves harbour without a destination to go to: They are more likely to get lost and find trouble.

Introduce the idea. The idea is just that. It is your direction – your purpose. A business does not progress unless it has some goal to progress towards, and, no, making lots of money is not a satisfactory goal to launch a business (although we all hope that we will be able to make lots – we will need something more concrete).

For me, my idea started more than a year ago, and its form was very different than the form I eventually ran with. The original idea was quite simple. I thought many companies were not handling their online reputation well and I thought that I could help them with damage control and bad reputation mitigation. I cited the BP oil spill as an example to my friends while trying to convince them that this idea was the best thing ever:

“Just think. Some company, like BP, has a huge reputation drop due to them doing something – like spilling millions of gallons into the ocean. Now whenever I Google ‘BP’ I get all that bad reputation on the front page: bad for business. What if I was able to provide consulting and help mitigate their online reputation damage.”

It sounded good. I started doing my market research and also looked into who else was working in this industry. Turns out that there were quite a few people who thought the same as I did.

My peers were skeptical. It seemed like a lot of manual work that would only succeed if I did all the work myself (Not really automatable). Not scalable at all. I would only be able to take on as many clients as I could handle myself – or hire lots of people to handle the clients for me.

I decided more discussion and bouncing ideas off of people was in order. I did this for almost six months before the notion hit me: All these sites assume that you know what your reputation is like! How do you quantify it? If I know exactly what it is, I can compare reputations!

It took another six months to generate a rough prototype to see if this was actually possible. Turns out it was! I was on my way!

 

The key lesson I learned here was quite simple: Really spend the time to understand the problem that you are trying to solve and make sure that people care about solving that particular problem. Understand what your goals are and make sure to express what the value of your solution is in terms of your client: why should your client care in other words. Also, note that the idea is malleable. It can change to what the customers actually want. It doesn’t help to make the best product in the world if nobody wants it.

How do you know if people will use a solution to that particular problem? Well, you could ask them. The simple solution is usually the easiest. In my case, I asked anyone who owned a company or was in mid-to-upper management of any company I could find. I asked them if they would be interested in a tool like mine and carefully listened to their responses.

Things were looking really good. I felt that I had a great product that was relevant to people in my target market. They seemed quite interested in it. It was time to look into this whole “starting a company” thing.

 

But how…

Standard