The role of Pen Testing / Vuln Hunting in Information Security

Intro

At a security conference, ask someone in attendance what they do.  More than likely they are a consultant, either doing penetration testing, vulnerability hunting or both.  Penetration testing and vulnerability hunting are mainstays of security testing, many times required by laws, regulations, or contracts.  They exist ubiquitously in information security.

But we don't have a good model for how they fit into improving defense.  The prevailing knowledge is that disclosing vulnerabilities leads to their mitigation which leads to more security.  However there is a counter-argument that disclosing vulnerabilities helps the attackers more than the defenders.  Can we build a model that takes both views into account?  Let's see.

So what do you 'do' here?

So what do penetration testers and vulnerability hunters actually 'do'?  If we think of information security as a game, (a very high-stakes game), we could say that penetration testers and vulnerability hunters reveal paths on the game board that attackers can take to reach their objectives.  That begs the question:

How does this benefit the defenders?

Let's take four scenarios:

1. No-one knows about the path:  In this case no-one benefits, no-one loses, because no-one knows. No change.
2. Only the defender knows about the path: In this case, the defender either benefits none or actually loses as they expend resources to mitigate the path. Defender Cost.
3. Both defender and attacker know about the path: In this case, the attacker either benefits some or none depending on whether they successfully exploit the path.  The defender probably loses some (mitigates the path) or loses a lot (is exploited) though there is the off chance they lose none due to the attacker's failed exploitation. Attacker potential Profit. Defender potential for more Cost.
4. Only the attacker knows about the path: Here the attacker's chance to benefit goes up significantly as the defender is unaware of the path.  The defender, on the other hand, doesn't even have the chance to mitigate the path and can only lose.  And after exploit, they return to step 3 and still lose as they mitigate the path. Attacker most Profit. Defender most Cost.

Conclusion

Based on the model above, penetration testers and vulnerability hunters can be most helpful by using their knowledge of paths to detect when attackers know them and to disclose them to defenders in situations when the attackers already know of the path.  This helps move from Scenario 4 to Scenario 3.  It's not ideal, but it's better than the status quo.

If only it were so simple

This model is admittedly naive.  It's a starting point, not an end-all-be-all.  Some things to consider:

• There is a time lag from knowledge of a path to its weaponization or mitigation.  The model should take that into account.
• Attackers and defenders are not homogenous.  This model doesn't consider what some attackers/defenders know and what others do not.  Nor does it model the spread of that knowledge through the population.
• This model relies on defender's knowledge of attacker's knowledge.  Something that will always be imperfect.
• Paths are made up of individual pieces.  This model doesn't account for the rearranging of pieces of the path, combined with other information in the attacker/defender's knowledge, to form new paths.

This model is not perfect, but hopefully it's a start in how to consider the role of penetration testing/vulnerability hunting in information security.

Alexi Hawk's Impossible Data Set

As the author of the only unsolved puzzle in the DBIR Cover Challenge this year, I figured I should provide a bit of a write up.  I'll apologize to all of the cover challenge participants as it's quite literally 10 lines of code to solve,  only two of which are actually functional (vs loading packages and naming stuff).

The Idea

First, where the puzzle came from.  I wanted to have a data-y puzzle in the challenge, but I also wanted it to be challenging for data science-y people.  To that end, I suggested, and the team approved, a puzzle based on a dataset, but with a twist.  The solution would not be from analyzing the data statistically.  Even then, our estimate going in was that it was the hardest puzzle of the bunch and likely wouldn't be solved.

The Setup

To create the puzzle, I used gimp to create a raster image with the key text.  I then opened the image in python using the PIL package.  It lets you parse through each of the individual pixels and determine its RGB.   I took all the pixels with RGB less than 10 (i.e. black) and saved them as a csv of (x, y) coordinates.

From there I transferred it to R.  Since each point is a pixel (i.e. closer than the size of a circle drawn at that location), I filtered down to 10% of the points.  Now, the first thing a good data scientist does is looks at the data, so we can't have it be that obvious.  Instead, I added a third column with random points in the range of the first two columns.  Then I swapped the first and third column.  If creating a scatter plot of the data would have been looking straight on, now doing so (on the first two columns) is like looking at the vertical location and a completely random horizontal location of the pixel.  

As we discussed the puzzle, someone else had suggested doing something with polar coordinates.  So I did just that.  I converted the current cartesian coordinates into spherical form.  (Hopefully all the hints about spheres and looking at the ranges now make sense as two columns, the angles in radians, range from about 0 to 1.6 and one, the vector length, ranges from 0 to about 500).

The Payout

So, the solution to the dataset (in R) is as follows:

# Read in the file

alexi <- read.csv("http://cybercdc.global/static/alexi.csv")

# Convert each from spherical coordinates to cartesian

library(pracma)

back <- apply(alexi, MARGIN=1, sph2cart) 

(At this point if you look at the data you'll notice two int rows and one numeric.  That wasn't intended and gives away the correct rows a bit.)

# The output of above is 1 point per column.  Change it to rows using the 't' (transpose) command. 

back <- t(back)

# Convert it back to a dataframe to make it easily plottable with ggplot

back <- as.data.frame(back)

# Give it column names to make it easy to refer to

names(back) <- c("V1", "V2", "V3")

# Scatter plot the correct two dimensions to view the data

library(ggplot2)

ggplot(as.data.frame(back)) + aes(x=V3, y=V2) + geom_point()

You may have to squish the vertical dimension a bit to read the text, but you'll see it.

Prologue

Incidentally, I actually tested that the spherical points to make sure that they wouldn't reveal the clue when visualized directly.  The sampling had to be adjusted so that if you visualized columns V1 and V3, it didn't reveal the activation text.

Hybrid Cybers

At the Women in Cyber Security Conference, someone posted a slide title "The Rise of the Cyber-Hybrid".  The concept was that to advance and develop in cyber security, people needed multiple disparate skills (policy, law, regulatory, interpersonal skills, leadership, etc).  While I don't disagree that having these skills makes someone more employable, I do disagree that they are a requirement.

Hiring a Cyber Hybrid

Instead, this is really more of a list of skills needed on a team in general. The higher up the org chart you go, the more of the skills are needed in aggregate.  As a hiring manager, it's easy request that full list of skills in a single employee for multiple reasons:

• It's easier to get approval to hire one unicorn than a technical person plus a social sciences major.
• You don't have to compromise.  The person has everything.
• If one person presents all the skills needed for the team, there's much less risk and work required to build the team.

However, while hiring the unicorn in theory sounds perfect, the practicality is far from it.

• The person with all skills, particularly disparate skills, are hard to find.  (And, let's face it, no-one is perfect.)
• When you do find them, they are both expensive and demanding.
• You may be hiring a hybrid who has multiple skills on the hope that since they are pretty good at everything, they'll figure out the role you need as well; just to find out that is not the case.
• If you do get them, they are hard to keep.
• Even if kept, unless all their skills are continuously utilized, they will lose some subset of them, which is a loss for both the employee and your organization.

Plus, they have an effect on the rest of the team.

• An entire team of unicorns is under-utilized.  If all are capable in all areas, yet you need 10 hours of technical work for each 1 hour of presenting, you are wasting a significant amount of the presenting skill on your team.
• Having one overachiever encourages the underachievers with overlapping skills to under perform.
• Having an overachiever that does everything can hurt morale for the rest of the team who have to work with someone who can do what they can do, plus all the things they can't.  It strongly encourages imposter syndrome.

The Alternative to Hiring a Cyber Hybrid

Instead of trying to hire a single baseball player that can play every position, take the Moneyball approach.  (Not that moneyball was anything more than good, common sense, team building.)  Spread the skills you need across your team.

First, you need to understand the skills you need.  Do you really need someone versed in international law?  (Maybe. Maybe not.)  Do you need a skilled communicator? (Almost absolutely yes.)  Build a matrix of the people you have and the skills you need and note who can provide what.  Once you know what skills you are lacking or weak in, plan to hire to obtain those skills.  That might mean adding an english major to your forensics team or building a strong relationship with an editor.  It may mean hiring a marketing guy who has great interpersonal skills to your pen test team to be the face to the customer.

Many times people have the team they have and simply can't go and add head-count to get the skills they need.  The reality is you probably have what you need in your team, it just takes some skill to tease it out.

1. First and foremost, care for the physical and emotional needs of your team.  I can't stress this enough.  If you don't, nothing else you do matters.  Your morale will be low and your team will underperform.  Everyone leaves for a reason and your team will leave you if their needs are not met.
2. Understand the strengths of your team members and maximizes them.  Find the guy who gets along well and use him for presenting.  Find the creative guy and have him suggest solutions to problems.  Find the thorough person and set them to checking technical details.   This is certainly harder than it sounds.  It takes a lot of talking with and observing your team and a lot of trial and error.  However, success is clear.  When the person's productivity shoots up, you know you've hit the nail on the head.
3. Compensate for team members' weaknesses.  The creative person will probably have bad ideas and miss details.  Get the pessimist's opinion on the ideas and let the detailed person check them.  The social person may not be highly technical.  Let them skip out on the high tech work.  The detail person may have trouble coming up with diverse ideas.  Don't put pressure on them to come up with solutions to hard, complex problems that the creative person can solve.
4. Continue to grow your team.  For the members who want to improve themselves, encourage them to go to management, interpersonal skill training (and make sure to save the budget to send them), technical training, or however they wish to expand.  Then give them responsibilities that allow them to practice what they've learened.  Many times the skills people are interested in gaining will become just as important as the skills they had when hired.

Benefits

There are a lot of benefits to this approach, (on top of not having the downsides listed above of hiring the hybrid).

• Your team is more likely to succeed, and succeed as a team.
• Each team members' skills are utilized.  No-one's skills, (which you pay for through differences in their negotiated salary), are going unused.
• Each individual team member is less expensive because you don't have to pay for skills you aren't using.  This also frees up money for additional training, which leads to the next bullet.
• Morale is higher.  Each team member is contributing in a substantive way.  Hopefully each member is happier due to better fit between their role and skills.  And hopefully you have more flexibility to grow your team members.

And, as you go, you are making the better-rounded people who are prepared to take the few roles where one person does need to have it all.  (These roles tend to be in small companies with 1-man teams or in management positions where the manager must have the social skills to deal up the chain and the technical skills to deal down it.)

For Employees

This isn't a blank check to be a one-trick pony.  You may very well be the best person in the country at reverse engineering malware, and that ability may get you the job you want.  But the gal with a reasonable amount of technical experience plus many soft skills and skills from other disciplines is probably the more desirable employee in most roles.  Instead,

• Make your boss's life easier.  The more of the skills in the WiCyS conference slide you possess, the more options they have in balancing their team.
• Also, the more skills you have, the more valuable you are to your current and future employers.  That means more compensation, more options, and more flexibility.

Conclusion

In the end, no-one's perfect.  You can hunt the cyber hybrid, but you're probably better off hiring imperfect people and building a team greater than any one person with the same skills.  And, as an employee, always work to build those additional skills to help your team.

Of Course the Network Diagrams are Bad!

As security professionals we know network diagrams are critical to providing security.  It's the top control in the SANS CIS CSC top 20 controls. Yet, almost every organization we go to has network diagrams that are convoluted, out-of-date, missing things, or just plain wrong.  Our pen tests produce better network diagrams than what the organization has in the span of the engagement.  Why is that?

1. Laying out networks is HARD!  Networks are really just graphs (in the mathematical sense) and graph layout consumes a lot of trees in the pursuit of academic publication.  Honestly, I have been laying graphs out for years, up to the 100's of 1,000's of nodes (where visualizations tools tend to top out) and  any graph with over just a dozen or two nodes is no longer self-explanatory.  Just look at Figure 15 in the PHIDBR.  It's pretty, but let's be honest; you can't really draw any conclusions simply by looking at the graph.
2. In reality, you'd need an artist.  Someone skilled in data visualization and with good artistic prowess to build useful network diagrams.  Yet, how's that going to work?  Do you hire an artist who knows your network?  Do you train your network guys in visual arts?  Do you hire a full time position simply to draw beautiful network diagrams?
3. And the network is always changing.  Those diagrams are likely to be obsolete as soon as they are completed.  Does the artist maintain them?  Do you hold back network changes for updates to the network diagram?
4. And even if you do get a good set of network diagrams that your artist-in-residence keeps up-to-date, what level of detail are you creating them at? Are you creating block diagrams that generally show the top level of the system in the abstract?  Are you creating wiring diagrams for the racks down to the power and ground cables?  Are you creating every potential view in the DODAF?  The reality is when people say "Show me the network diagram." what they mean is "Show me the network diagram showing me exactly the things I'm interested in at the level of detail that I think is correct but that I have never communicated."
5. And none of this even begins to touch on the issue of determining ground-truth in how your network is connected.  It's hard when you know all the devices and can dump all the configurations.  It's damn near impossible in a practical network when people add things without saying, and use equipment that is not centrally managed.

The reality is it'd be surprising to find someone keeping great network diagrams, simply because of the amount of effort involved.  There are automated tools to help, but if a human can't easily make the network visually understandable, the software is not going to do better.  Also, the software suffers from the same problems related to level-of-detail, being up-to-date, and accurately discovering the true network that a person manually doing the job would.

So are there solutions?  I don't know.  Probably not.  I think that a real-time interactive visualization system rather than static pieces of paper is better.  A system designed with a certain amount of artificial intelligence to learn and explore the network would probably help.  However, we simply may need to accept that we won't know our network fully and that the situation is more like:

*

Knowing this, sympathize with organizations doing their best, and help plan a defense that accepts this reality.

* I saw this on twitter but can't find it again.  If anyone has proper attribution I'd be happy to add it.

Is Your Objective Risk Assessment Methodology Really Objective? Really?

Introduction

I hear a lot about Risk Assessment Methodologies (RAMs) and making risk assessment objective these days.  Let me pass on some lessons learned in a previous attempt to make risk objective.

Bucketing

Most organizations that attempt to make risk objective do what I affectionately call 'bucketing'.  This is when you create buckets which define a risk and then assign them values. For example, here are the buckets for the Common Vulnerability Scoring System, (CVSS), from First:

You may use other values such as whether a component is often used or what team runs it.  I call this Bucketing.  Risks fall like raindrops all over the map and you are setting out buckets to try to make sure each risk falls into one that you can give it a score from.

The Gotchas with Bucketing

Bucketing appears very alluring, but in practice it simply does not work the way it's creators expected it to.  Like all good security professionals, the users of the RAM game it to their own ends.  The list below represents some of the tricks that occur in RAMs based on bucketing.

1. All Scores the same. See Michael Roytman's talk where he discusses issues with CVEs.  Especially pay attention to the issues with separation of scores.  If you use some form of bucketing for your RAM, you'll have a lot of options, but in practice you'll only use a few of them, which will mean you'll only get a few unique scores.
2. No matter how many buckets, you'll never have enough.  The reason you'll have those unused buckets is the rain problem. No matter how many buckets you set out, you'll never capture every rain drop.  How that applies here is fairly straightforward.  You will start with a set of buckets such as those in CVSS.  But you will find out some risks are falling between the buckets, so you will make more, smaller, buckets.  Then you will have so many buckets that no-one can remember how they are different, so you'll combine buckets.  And no matter how many times you split or combine buckets, risks will always fall through the cracks between them.
3. The JG Memorial Fire Axe.  This is a story of relative risk.  I was once told a power button on a mainframe was a HIGH RISK if the door to the mainframe wasn't locked.  However, an astute engineer pointed out that there was a fire axe on the wall and he could just as easily cut the cables.  In fact, anyone with physical access had ample ability to cause the mainframe to fail.  The point was that it was not the absolute risk of having a power button, but the relative change in aggregate risk that the risk causes.  Bucketing systems are simply not designed to handle the interplay of multiple risks.
4. Back-Engineering.  This is the biggest problem with Bucketing.  The reality is you have a risk analyst in the process.  No matter how objective you make the rest of it, the analyst will look at the risk and immediately decide how significant a risk it is.  From there, if the buckets they assigned the risk don't add up to what they want, (such as getting a 'low' score for a risk they thought should be 'high'), they'll simply change them to something a that could still be true, but makes the score what they want it to be.  After getting some experience with the RAM, they will get very good at back-engineering; to the point where every risk comes out with the score they want it to have, not the score the RAM originally suggested for it.
5. Hypotheticals.  What enables Trick 4 is the fact that the score represents more than just an atomic risk.  Instead, it represents an entire context.  Take for example a SQL injection in a webapp.  That simply doesn't tell you enough to understand the risk, instead you have to make decisions about how easily it could be exploited, what it's exposure is, etc, etc.  The analysts assigning the risks may discuss and decide that the SQLi is really hard to access due to it being blind with no debug information.  Another analyst may say "well, but if they knew the entire DB schema, it would be easy to exploit and we can't prove they don't know the DB schema" so the group ranks it "easy" to exploit and it is listed as a 'high' risk.  This leads trick 6.
6. No Documentation.  Because the buckets are self-documenting, right?  No need to write down that a hypothetical discussion changed the risk.  No need to capture that it came out a 'low' risk in the first analysis.  The buckets documented the risk and, therefore, no additional context need be documented.  This almost ensures the score will not be repeatable.  It does, however, suggest a better way.

Other Options: Capturing Context

The good news is fixing this is a single word: CONTEXT.  Almost all of the problems stem from lack of a documentation of the context of the risk.  To alleviate them, you simply need to document the context.  The easiest way is to write down, in narrative form, all the steps you see happening in the exploitation of the risk, what the impact would be, and assumptions you've made.  Something like:

The attacker decides they want to attack us.  They've watched youtube videos on hacking and have downloaded Kali, but not much else.  They run a scanner against our webapp which returns the login and password in comments in the code.  They login and copy and paste a SQLi into the DB query form on the admin page which returns a file with the entire database.  They take it and post it on pastebin.

My experience has been that, once you fully document the context, how everyone is thinking of the risk, subjectives scores tend to be the same.  As such you could simply ask your analysts to score the risk on a scale of 0-<TOP> where 0 is cannot happen/no impact and <TOP> is will happen/the greatest impact they can think of.

Another approach to consider is the Thomas Scoring System.  Russell Thomas has put a large amount of work into it and it is a very good way of not only capturing context but also linking that context to your score.  You can watch the video explaining it as well as read the blog or just download the tool he's created!  (I'd recommend watching the video, or at least skimming it, then downloading the tool.)

And there are even ways to improve on capturing context, however we'll leave those for another blog.

It Could Be Worse Than Bucketing

And though this post isn't about them, there are things even worse than Bucketing.  "Not Tracking Risk" for example.  Or using the tool report as your risk report.   Taking what your vulnerability scanner told you carte blanche is the quickest way to have leadership ignore you when you bring in the 1500 page report listing 10,000 high risks.

Conclusion

In the end, the push towards objective risk management is a good thing.  That said, we have a long way to go.  If you make it to Bucketing, good for you.  It's a step in the right direction.  But don't consider the job done.  You're much better off taking the next, small, step to risk based on context!

Apprenticeship and Infosec

So how do you learn to be an infosec professional? Honestly, most of the leaders in the field these days were the stuckie, (i.e. the guy who didn't say "not it" quick enough), in the office when a security person was needed. While infosec academic programs exist, the reality is almost all security positions require experience. As a friend put it on Twitter, "Schools can do a great job on teaching technology, but methodology and process require more than book knowledge." Now, even strategy and process can be taught academically as military and criminal justice degrees show. But the reality is, even after completing basic training or your criminal justice degree, you're still a rookie.

The reality is infosec is much more like a traditional profession than newer, technical, professions.  And most traditional professions are based around apprenticeship.  If you were going to DO something in the old days, you didn't learn it so much in the university as the back of a current practitioner's shop.

And that's still the case in many careers.  A welder who can weld what others cannot is valuable.  Air traffic controllers have a median pay of $122k/year.  Even in highly educated careers, the education is really just an introduction to the on-the-job training.  Medical doctors have residencies.  Engineers must study under a Professional Engineer to become one.  Teachers student teach.  Nurses precept.

So what about information security?  What really needs to be taught in a classroom?  Probably the basic controls and technology, though not in any depth as it'll have changed by the time the student enters the field anyway.  Probably general strategies and some basic processes.  After that though, why is there not a formal, controlled, apprenticeship process for information security as there is for so many other fields?  Why do infosec students not practice engineering security, working incidents, and gathering intelligence the same way doctors practice internal medicine, surgery, and triage medicine?  We all know the apprenticeship is happening one way or another, so why not formalize it? 

Marisa Fagan suggested a mentorship program almost half a decade ago and not much has changed since then.  Still, we've all matured a bit.  We now understand the importance of working with the large, existing institutions where we used to go it alone.  Maybe it's time to make apprenticeship an expected and formally defined part of the information security curriculum.

Make Your Own DBIR Charts!

Today starts a four blog series on making your own DBIR charts.  Learn about the tools and processes to go from your excel file of incidents to the charts in the Verizon DBIR!

• Make Your Own DBIR Charts! Part 1 of 4: The tools!
• Make Your Own DBIR Charts! Part 2 of 4: Making VERIS Data
• Make Your Own DBIR Charts! Part 3 of 4: From JSON to Charts!
• Make Your Own DBIR Charts! Part 4 of 4: See Trends over Time