> The National Security Agency (NSA) is the font of information security wisdom for the US defense and intelligence communities. But apparently, the NSA's own network security is so weak that a single administrator was able to hijack the credentials of a number of NSA employees with high-level security clearances and use them to download data from the agency's internal networks.
Thats a bit hyperbolic and out of touch with reality. Sure I as a sysadmin with root to most UNIX machines in my companies environment could have been able to copy the raw Oracle db files to steal company secrets, SAP databases for other juicy data that I could sell to a competitor, run a network sniffer on important login servers to steal passwords, that is how the real world works. If anyone believes that you can totally lock down access to every system on your network from your trusted sysadmins and have 100% audibility and accountability you are unfortunately living in a fantasy land. NSA or not, this really isn't something that is 100% preventable.
The solution, which they're implementing now (http://www.forbes.com/sites/andygreenberg/2013/06/18/nsa-dir...), is to require two or more administrators present whenever anyone has to build, maintain, or otherwise access potentially sensitive software, computers, information, etc.
They would be required to report on any strange activity they see from the other administrator(s).
Extremely inefficient, but it's far more secure than just allowing any administrator to do what they want with stuff.
The more secretive or unjust an organization is, the more leaks induce
fear and paranoia in its leadership and planning coterie. This must
result in minimization of efficient internal communications
mechanisms (an increase in cognitive "secrecy tax") and consequent
system-wide cognitive decline resulting in decreased ability
to hold onto power as the environment demands adaption.
Not a bad concept. But your accountabil-a-buddy has to be at the same level as you technically. One can easily fool or mislead non-technical people. "Q: What are you doing?" "A: Rebooting the flux capacitor"
I've worked (in non-government contexts) under rules that required two people for certain actions.
My experiences with such processes is that unless you can have some sort of technical measure that proves both people are actually paying active attention to what is being done, the second person will often just zone out. Sometimes, if asked even a few hours later, they won't even have a clear recollection of the event taking place, much less what was actually done.
Additionally if you both double up and cut the workforce, chances are the two will be doubling up in another way too, i.e. both working on different tasks and claiming to keep an eye on eachother.
But really, you are going to lay off 90% of the sysadmins and require two different people involved to change a password?
What this shows is if anything how much you need a combination of good monitoring and enough people. And once one account is compromised you have a chance for the sysadmin to be using sock puppets for accountability actions.
Is it? I'd expect a write-only logging system that runs at a low level exporting logging to remote servers that other admins don't have access to. As the article suggests, you layer admins so no single person can truly compromise things.
It should be pretty basic to catch someone that's resetting user's passwords.
Sure, you can remote syslog everything to a remote logging service, but:
1] Do you log every single thing that happens on the machine?
2] If not, did you actually log the proper set of commands that were used to possibly commit a crime?
3] Granted you are not a small startup, do you have a process in place already to data mine, extract and analyze these potentially dangerous commands from log sets that are potentially TB in size?
4] Do you have the personnel in place with the qualifications and time each day/week/month to audit these potentially large results?
5] If you've caught something, aren't you already too late? Do you have some type of magical software in place that will somehow recognize these types of crimes in progress and cut off access in realtime?
Many of these articles totally ignore how these real world systems work (mainly I'm guessing because the authors have never been involved with companies that run 10k-200k servers around the world). It really is more complex than you think. Most companies that aren't startup size are continually playing catch up. Sure they need to put logging in place, auditing, yada yada. That is being balanced with the other day to day pressing tasks also. It always seems to be a game of catch up...
1+2) You could catch a decent portion of this type of behaviour with a tiny fraction of the effort. Logging local activity on individual user desktops is a waste of time, for example. Raising the difficulty of doing things undetected raises the bar significantly, particularly if users don't know exactly what's going to trip an alarm somewhere and get them caught.
3+4) Considering the nature of their work, I'd expect the NSA to be the right people for this with existing processes, suitable personnel and the storage/processing power required. It's their job to intercept and analyse large quantities of network traffic.
Yes, it's a difficult problem (and impossible to comprehensively solve, doubly so when you don't trust administrator-level users), but when you're dealing with classified information in government, security is more important than in a regular commercial context.
If anything, the lack of decent internal auditing at the NSA is probably intentional.
At last year's AWS Public Sector Summit, I watched a demo of Xceedium's access control appliance [1]. It's deployed as a jump host and acts like a DVR for SSH/RDP. The CAC crowd was very interested.
Re #1 – at the rate the NSA is reported to be archiving data, do you _really_ think logging every single keystroke that every employee/contractor with any elevated privileges at all ever types would be even vaguely a problem? TB-sized logfiles and processes to datamine them are surely interview tasks or intern familiarisation exercises at their scale?
Yes. A previous employer of mine was apparently logging a PB of log data a day between all departments, it isn't something that is just tossed up and somehow works in a month.
Do you have infrastructure in place to store all of that? How are you storing all of this content coherently so that you can go back and audit records of systems at a point in time you are concerned about? Do you have an internal Hadoop cluster configured (not trivial) and the hardware to ingest and process these logs? How are you actually logging everything? Do you have dedicated and qualified people to write this software and maintain it? When someone has su -'d to root, how are you logging everything they type coherently? Is your regex smart enough to ingest and contextualize multiline commands as root and add them to your report? What if no actual "bad" commands on your list were typed as root? What if I as a luser copied the "bad" commands to /tmp/kittens.txt, then su -'d to root and ran them there? Is this somehow captured? Are you 100% sure syslog is running 100% of the time on all hosts you are concerned about? Can't I as a sysadmin kill the syslog pid before I commit a crime? Are you using redundant syslog hosts? Are you using TCP and not the default UDP so that syslog doesn't drop packets under load?
Like I said, this isn't as simple as everyone thinks...
Ballpark numbers - an employee typing at 100wpm for 40 hours a week generates not quite 70MB worth of keystrokes in a year - and I'd guess that estimate is probably something like 2 orders of magnitude too high - no-one actually types 100wpm nonstop all day every single day at work. Even ignoring that, I've got enough disk space sitting under my TV to store (uncompressed) every keystroke typed in a year by something like 100,000 such mythical "100wpm typists working at 100% utilisation" employees. 1PB would, to a first approximation store every single keypress it'd be possible for the rumored 4 million "top secret or above clearance" people in the US typing flat out for something like 15 years.
So I don't think your "do you have the infrastructure" argument holds too much water here.
The "do you have internal Hadoop clusters" question falls the same way. Sure, I don't have that connected to my 8TB of external USB drives plugged into my media server - but if I worked at Google or FaceBook or Twitter, I'd fully expect to be able to provision and spin up adequately sized clusters of VMs and storage to effectively consume and run reports on data that size and bigger. And it's surely not just the NSA and Google/Facebook/Twitter routinely dealing with collecting and processing data at that scale - any decent sized telco, any non-trivial web analytics service, any large financial institution, every HFT business, most bio-med businesses, probably every physics and astronomy department at any university – there must be tens of thousands of businesses routinely dealing with that sort of sized data sets.
It's not simple - certainly not simple enough for me to do it on a Mac Mini and a bunch of external hard drives – but I also don't think it's anything like "uncharted waters" territory. (I'm pretty sure I could find the expertise required in my 1st level LinkedIn connections, and have absolutely no doubt I'd be able to manage designing, developing and deploying exactly such a system if someone came to me with a high six or low seven figure budget.)
I think your approximations are focusing solely on keystrokes, whereas the parent specified just "data", which makes me believe that network traffic, application logs, etc are included in this. I can believe the 1PB/day number, it's not that far fetched when you consider the above.
I also agree with the parent. There are so many possible scenarios and things to log that eventually you're playing a "logging" version of whack-a-mole. Even just managing these files (as the parent talks about) is really no trivial task. Honestly, I wouldn't even know how to begin managing a petabyte worth of daily data.
Spying on all of the email isn't simple either. When there's a will and a bottomless font of taxpayer money in the name of national security, there's a way.
First, you're tossing 100% around way too much - security in this environment is a pretty huge sliding scale. But for something as basic as password resets, yes, I'd expect 100% functionality and auditing to catch it.
Second, I don't think anyone thinks it's totally trivial. But, this is the NSA we're discussing. Supposedly, their secrets are so special, the USA will collapse or something if they're compromised. If any organization in the world is set to handle an admin resetting people's passwords, it's the NSA.
Are you really arguing that it's just too hard for the NSA to notice a massive violation of policy?
Sysadmins are supposed to reset passwords...or I'm sure there is some central auth service in place that allows access to hosts one has been determined to have access to.
The people you trust to admin systems are going to easily be able to abuse their power and it is very hard to stop them from doing so, without making their job so cumbersome for it to be near impossible.
I worked at a firm in 1999 as IT manager... the director required our syslog to be piped out to a line printer printing a real-time physical copy of log/messages and logwatch output on our linux machines.
I was required to manually go through each 24-hour output with a highlighter and look for attacks....
I had systems doing this in the late '90s, and I'm 100% certain I was following "current best practice" at the time, rather than having invented anything novel or notable…
(Having sid that, I wonder how many people have patents or patent applications in for that idea right now?)
>If anyone believes that you can totally lock down access to every system on your network from your trusted sysadmins and have 100% audibility and accountability you are unfortunately living in a fantasy land. NSA or not, this really isn't something that is 100% preventable.
A /huge/ part of managing systems is vetting the folks who must be completely trusted. As part of this, you want to reduce the number of people you trust 100%, but yeah, some still exist.
Those people you trust 100%? you must vet them carefully. You should not put some random contractor in that position. Clearly, as Regan would say, "mistakes were made" - and nobody seems to be taking responsibility.
Just keeping the secret data on a secured server that only a few, very highly vetted sysadmins had root access to, that carefully logged all requests for information (and set off a pager somewhere if too many unscheduled requests were made) would have solved the problem, assuming you didn't hire some random body shop to staff /those/ sysadmins.
I mean hell, I get paged sometimes because some fuckwit at one of my upstreams starts bouncing my packets that are traveling from san jose to sacramento through texas. Surely, someone could be woken up if someone starts accessing suspicious amounts of data at once.
This is part of the huge "who you know" factor in the valley; Generally speaking, you hire folks that your current folks know. Not only does this provide some technical validation, it also makes the cost of defection higher. (Of course, there are lots of downsides to that approach, too.)
But no matter how you do your vetting, you /must/ vet folks with root.
And you can (and should) reduce the number of folks with full root. Especially when there is sensitive data on hand. Give your contractors limited tools.
> Just keeping the secret data on a secured server that only a few, very highly vetted sysadmins had root access to, that carefully logged all requests for information (and set off a pager somewhere if too many unscheduled requests were made) would have solved the problem, assuming you didn't hire some random body shop to staff /those/ sysadmins.
The problem with that theory is that you're assuming only some data is secret. Actually, all the data is secret. Even the information about which data is secret, or what are the criteria of secret data, or how secret data should be handled. Everything is secret.
So rather than dealing with a neat pack of documents that you want to keep secret, think of an organisation with 100k+ people where every single bit of data they produce or interact with every single day is top secret.
Snowden's escape from the NSA exposed two things: 1) illegal kleptocratic behaviour on part of the government 2) gross security incompetence on part of the government. The NSA has spent all its time on #1 up until now, so they're going to hype him up as much as possible so as to diminish the embarrassment from #2.
> He wasn't just a community college stooge, he was brilliant! The obscure flaw that he exploited has since been fixed, hooray too!
Meanwhile 'sudo su' has been criminalized as a precaution.
sudo vim means vim gets root privileges. Meaning vim can do anything it wants if it's bugged or somehow compromised. Not likely on an otherwie secure system but not impossible.
sudoedit (according to the manpage) makes a copy of the file first, lets your editor edit it, then copies back when you're done -- so even a compromised editor couldn't do much damage beyond corrupting the given file.
And this is one of the reasons why the argument 'got nothing to hide' doesn't count.
When a hacker can access all this information it becomes easy to steal someones identity.
That's not entirely true. Take for example, passwords. If your requirement is "store passwords securely, so they can only be used to check logins", then you implement that as salted, one-way encryption. But if you add the shadow requirement "and be able to retrieve anyone's password when the government requests it", then your password storage will inevitably be much more vulnerable.
Agreed, data not stored is more secure than data that's stored, but I stand by my point that data stored in such a way as to provide the minimal necessary access can be substantially more secure than data stored in such a way as to provide that access, plus additional back-door access for spooks.
Personal observation:
Internal security best practices in spy organizations are rarely 'overlooked'. It is all about trade-offs. II think the more important question is "why would the NSA have lax internal oversight on both user-privledges AND audit-logs?"
The answer is that it is much easier for black bag operations to be scrubbed from potential oversight when an individual holds the power to run the hidden|illegal analysis and clean their own log trails.
Spring clean time at the NSA? Apart from anything he might have done, Snowden has provided a convenient dumpster, which may be used to neatly wrap up and dispose of pesky unexplained incidents.
"General, these logs say your account pulled the intercepts for a bunch of corporate board meetings, everyone within two hops of Senator Wyden, and General Petraeus. That was all just brilliant superspy Snowden, right?"
AKA he typed "su". Or whatever the Windows equivalent is. I do this all the time to diagnose problems and no one has ever written an article about me. :/
FWIW, there are some things that lots of sysadmins do on a week-to-week basis as part of their job – that other people have ended up with felony convictions for…
(Note: all 3 felony convictions were "expunged" eventually, but for 12 years he had all the restrictions and problems a convicted felon lives with, all for "doing his job", or possibly "overstepping the bounds of his authority while doing his job".)
If they couldn't detect Snowden, it raises the question of what the Chinese government has access to. I imagine every government and hacking group in the world is doing everything they can to get access to that pile of data. Even if you trust the US government not to abuse these capabilities, what will happen when that data falls into even worse hands?
I think it's a US thing though. Obviously I know that in US English the word has both meanings, but in the UK a troll is something that hides under a bridge, and the fishing is trawling.
Took me a while to figure out the humour inherent in the double meaning when I first came across it (in the 90s)
This appears to be a snowjob against Snowden, and BS -- Look at the language:
1. Snowden impersonated NSA officials, sources say
2. Edward Snowden accessed some secret national security documents by assuming the electronic identities of top NSA officials
3. forensic investigation has included trying to figure out which higher level officials Snowden impersonated
4. if an employee was on vacation while the on-line version of the employee was downloading a classified document, it might indicate that someone assumed the employee’s identity
5. NSA has already identified several instances where Snowden borrowed someone else’s user profile to access documents
6. “The damage, on a scale of 1 to 10, is a 12,” said a former intelligence official.
7. The NSA declined to comment <--- WTF, then who are the above sources?
[Edit: I wanted to add a little bit of clarity here: the language used is very vague and references things that could never possible be confirmed: sources say, "might indicate", "has identified" --- This story is like a bunch of paragraphs typed out, randomly put into a hat then shaken onto the floor into the pattern of the story. It is not a decisive, cohesive piece of information -- then it is ended saying that the NSA has no comment.
THe TITLE is "NSA finds Snowden hijacked officials’ logins" NSA FINDS....
So, if the NSA doesn't comment - and the "analysis by NBC" and the NSA declines to comment are all used -- then NOTHING in this piece can be believed.
Even if the entire premise is true - this is hands down the worst framing of the information, supposedly factual, one could imagine!
---
In my informed IT professional opinion, they are using this to brand him a hacker - and they make a bunch of "what if" type claims. Then they slide into a confirmed report. Then they claim the damage is off the scale (12 on a scale of 1-10)
This is a completely MISO built PR piece for the NSA.
As administrator on any system (administrator in Windows, and Root in *nix) one will have access to whatever you want.
Whilst at lockheed, I had admin rights to every machine and document in my realm - I would have had no need to "impersonate" any other lockheed employee...
The mistake here is if NSA was using the same root passwords/keys across entire tiers of machines. In that case - call it criminal negligence on the part of whomever architected that disaster.
My thoughts exactly, they are setting him up for CFAA charges either to pile up on him or so they can drop espionage taking away the reasons for Russia's asylum.
But, yes - it is pretty clear that they have the MSM on their freaking sqwak-box right now against Snowden.
Funny how it was also revealed that they (the UK) were lying about being "leaked" info from Snowden to the "Independent" -- where Snowden came out and said he never communicated to them anything, and the "leak" was a lie and information that Snowden was specifically avoiding getting out because it was too pointed at actual personnel...
This info needs to KEEP COMING -- as we need to get to a tipping point where change is a reality.
The "budget PDF" referenced "media exploitation" and my first thought was manipulating the mainstream media into basically reporting what they wanted and if some of the taxpayer funds given to the NSA goes into the pockets of various reporters and journalists.
Maybe the "media exploitation" is what you mentioned in your original post; in effect, "NSA officials", "sources inside the NSA", etc. all "anonymously" making off-the-record comments to journalists.
If the biggest story in the history of the NSA is that some college drop-out "genius" was able to exfiltrate 20,000 docs from the NSA - and the biggest key argument they have against the guy, to prove that he were some "hacker" who was impersonating people to gain access to said files -- how is it that "NBC" is revealing this with "sources say" "an official" etc...
Don't you think that the most critical point to the credibility of the NSA is to come out and state via an official means - that they have electronic logs of this activity?
Digital forensics are binary: Either you have a digital trace of actions taken, or you do not. Period.
Either they can determine 100% that Snowden logged into station X as person Y to grab file Z -- or they do not.
THe language used in this article is textbook MISO/Psyop PR.
"Oh, an official source with knowledge of the incident has said.." -- cool, I guess they have it figured out then, huh! I shouldn't dare question that. But if I do: "We can't comment" "we cannot reveal the details of an ongoing investigation" etc...
To me this simply looks like the folks in charge are locking in their cyber crimes case against Snowden. If he's ever brought in--for whatever reason--and even if he magically avoids every charge of espionage, treason, leaking, spying, misuse of company keyboards, or whatever, they'll have the hacking angle sunk so deep that it won't matter.
If Bradley Manning got what he did for scary wget wizardry (making no statement about the validity of that charge or verdict) then I think Snowden can safely expect more consecutive life sentences than he has fingers and toes.
Has anyone figured out exactly how high Snowden's clearance went?
They're really non-specific about what he did (and play it off like he couldn't do anything), but it's coming across more & more like he really had his crap together.
Thats a bit hyperbolic and out of touch with reality. Sure I as a sysadmin with root to most UNIX machines in my companies environment could have been able to copy the raw Oracle db files to steal company secrets, SAP databases for other juicy data that I could sell to a competitor, run a network sniffer on important login servers to steal passwords, that is how the real world works. If anyone believes that you can totally lock down access to every system on your network from your trusted sysadmins and have 100% audibility and accountability you are unfortunately living in a fantasy land. NSA or not, this really isn't something that is 100% preventable.