O
oddball6a
Guest
I have been trying very hard not to rant about this subject this week. Very hard indeed. My own suppressive instincts have come to thier end so I will write on this subject once, and hopefully in a place that it may accomplish something constructive.
For the past week, we in the general gaming community have heard from countless sources including Gabe himself that Half Life 2 source code was leaked. Some of us (being the curious, sometimes malicious souls that we are) have actually gotten our grubby little paws on the code, even if many of those who have it could never understand it. At first, we all felt shock, i mean who had ever heard of an event like this striking a game developer this openly before? And then incredulity, i mean its not like its every day that a Half Life 2-caliber project (arguably one of the highest-profile, most-anticipated projects in the history of electronic gaming) has a security breach of this nature. And finally, curiosity and anger roll in. Who would be willing to interfere with such a project? Who would be willing to risk exposing themselves to criminal sanctions in order to get a bit of source code? Why? And, most importantly, how the heck could they do it? I mean, a huge developer like valve with such a high profile project HAS to have great security, right?
Sorry, folks, the unfortunate reality is that all is not what it seems.
Having been a gamer most of my life, I know that most of us expect that game development companies know what they are doing all the time. We dont have to worry about the stupid realities of commercial business infringing on our awesome gaming projects. Who cares about corporate espionage until it strikes close to home? We automatically assume that our favorite gaming house has these bases covered, that they know how to take care of thier network and thier assets. Because of our own fan-boy statuses and the high profile nature of the project, we ascribe virtues of safety, security, and such to a company's resources without really knowing.
The last year and a half, as I have gradually come inside the industry to work Information Technology has really been an Eye opener for me. From what I understand and can sift out of reports, and what I know about the industry today, I can tell you that it may not be entirely the fault of the grunt day-to-day IT guys in the Network Operations Center, the Network Administrator's cubicle, or the Datacenters which are entirely to blame. Indeed, some of this security breach must ultimately be laid at the feet of management and the tendency for Game Developers to overlook internal security.
Most game developers these days employ a (relatively) simple networking model where there are a few points of external access. On these points of access, there may be security devices in the way of state-based packet inspection algorythms, bastion hosts to handle individual functions (mail proxy server, HTTP proxy, instant message proxy, etc), protocol-based directives (certain traffic is restricted to certain network segments or machines), and Network Address Translation (using non-publicly-routable IP addressing for internal infrastructure) to protect the internal network from external harm. Few companies these days pay particular attention to securing individual development teams, the myriad internal computing resources updates, or penetration detection within a network. Fewer still actually dedicate IT resources to network security.
Im not saying that this is criminal or that I think that people who have this problem are idiots, I am simply saying that it IS a problem which is often not seen until it bites people in the delicate rear such as we see here. I mentioned a minute ago that the IT professionals which do the actual day to day work may not be entirely to blame. Most game development firms dedicate a very small expenditure to IT Departments because the IT department is probably one of the worst capital costs for a firm from which it is guarenteed there will be little or no revenue stream unless the company plans on offering application hosting to business partners or investing in a similar service not central to the corporate product line. This lower level of resource spending means that some places have to be cut short. It may mean 1 security analyst instead of 2 or 3. It may mean that the Network Administrator or Internetworking professional has to take over non-dedicated security duties in addition to primary function. This tends to compromise company security. Further, software nowadays is so difficult to patch and keep current particularly accross many desktops that you can scan a particular network and find at least a 50% difference accross your enterprise infrastructure in versioning of desktops, productivity software, and internet related applications. The aforementioned lack of resources means that companies often cant dedicated IT support time to updating machines accross the enterprise or deploying a form of Operating System/Application version management.
So, we end up with situations like the one that Half Life has been hurt by. An outdated application (Microsoft Outlook) was installed on a sensitive machine (apparently Gabe's) which had direct access to sensitive company information (the Half Life 2 source tree). There was no internal security on the source tree that we know of. There was no internal security on the machine that we are aware of. There was no segregation or sequestering of Developing network segments. And finally, it is apparent there was no version control software for keeping sensitive machines up to date. These multiple faults all contributed to the success of the recent attack and the compromise of sensitive information.
I, for one, hope this not only serves to make Valve aware of the need for information technology security, but between this attack, the recent Email worms, and blaster, more enterprises and particularly game developers take heed of recent events.
How do I think this could have been prevented? Implement multilayered security. Try to do some security forecasting for major projects. Allow a larger margin to be spent on IT resources, it pays off in the long run in avoiding events like this even if no revenue from these expeditures are showing on the balance sheet. Isolate development partially.
Ok, first, the external security. Use enterprise-level network address translation to segregate your internal infrastructure from the internet. Use protocol based proxying. Put high profile externally accessible machines on a DMZ. Use active intrusion detection (cisco makes good software for this, checkpoint used to be pretty good but are now falling slightly behind in terms of implemented technology.)
Preplan your network model. With NAT, you can build a network structure you need. Spend the extra money on layer 3 switches and build a flexible network which assigns entire ranges of class C addresses to individual company functions. Remember that using NAT with the 192.168 range allows you to use 2^16 addresses since the range is Class B. You have plenty of address space at little to no cost, so spread it around. Management should have thier own subnet. Accounting/finance should have thier own subnet. Each Development team should have thier own subnet. Common Corporate infrastructure should be on its own "server" subnet. This allows more flexibility with routing and securing access between corporate areas. Yes, it takes a little more to set up, and some foresight, but it makes events like this more difficult by segregating access points. Compromise a machine in finance and there shouldnt be any danger to development teams or development servers.
Many people in the past week have been running around "why werent the developers on a LAN"? Well, most game companies are not willing to accept the loss of productivity and the annoyance that complete segregation entails. What could have been done (since theoretically our developers are on thier own network segment, RIGHT?) is implement a segment-specific proxying and filtering. Certain common network assets are accessible, very little else can get in or out inside the company. No one that is not DIRECTLY related to that project's development has the ability to penetrate the network isolation. However developers still have internet access, can still email, and still have a reasonable level of security.
Implement product version management. There are online services to assist with this as well as stand-alone internal network products which work as a client-server system, downloading updates to a central server and then deploying them internally to company machines. Something as simple as this could have saved all this time and trouble. When gabe connected into the network, the version management sensed the network connection, requested the latest update check from the central server, and just like that outlook security fixes are patched and gabe is asked to please reboot.
There is no silver bullet, no perfect security solution short of an Adaptive Packet Destruction Filter (a pair of wire cutters for those of you not familiar with internetworking). People who are absolutely determined will eventually penetrate something. Its the job of IT and IT management to limit that exposure and make the network penetration as difficult as possible with as little impact as the IT department can possibly get away with.
This wont affect game sales but it definitely cost Valve some reputation, a huge dose of embarassment, and put thier shader routines into the wild. Valve failed this time. Dont fire your IT grunts. Hire a couple new ones. Buy some version management software, plan for an infrastructure rebuild after this game releases. Make sure it doesnt happen again.
For the past week, we in the general gaming community have heard from countless sources including Gabe himself that Half Life 2 source code was leaked. Some of us (being the curious, sometimes malicious souls that we are) have actually gotten our grubby little paws on the code, even if many of those who have it could never understand it. At first, we all felt shock, i mean who had ever heard of an event like this striking a game developer this openly before? And then incredulity, i mean its not like its every day that a Half Life 2-caliber project (arguably one of the highest-profile, most-anticipated projects in the history of electronic gaming) has a security breach of this nature. And finally, curiosity and anger roll in. Who would be willing to interfere with such a project? Who would be willing to risk exposing themselves to criminal sanctions in order to get a bit of source code? Why? And, most importantly, how the heck could they do it? I mean, a huge developer like valve with such a high profile project HAS to have great security, right?
Sorry, folks, the unfortunate reality is that all is not what it seems.
Having been a gamer most of my life, I know that most of us expect that game development companies know what they are doing all the time. We dont have to worry about the stupid realities of commercial business infringing on our awesome gaming projects. Who cares about corporate espionage until it strikes close to home? We automatically assume that our favorite gaming house has these bases covered, that they know how to take care of thier network and thier assets. Because of our own fan-boy statuses and the high profile nature of the project, we ascribe virtues of safety, security, and such to a company's resources without really knowing.
The last year and a half, as I have gradually come inside the industry to work Information Technology has really been an Eye opener for me. From what I understand and can sift out of reports, and what I know about the industry today, I can tell you that it may not be entirely the fault of the grunt day-to-day IT guys in the Network Operations Center, the Network Administrator's cubicle, or the Datacenters which are entirely to blame. Indeed, some of this security breach must ultimately be laid at the feet of management and the tendency for Game Developers to overlook internal security.
Most game developers these days employ a (relatively) simple networking model where there are a few points of external access. On these points of access, there may be security devices in the way of state-based packet inspection algorythms, bastion hosts to handle individual functions (mail proxy server, HTTP proxy, instant message proxy, etc), protocol-based directives (certain traffic is restricted to certain network segments or machines), and Network Address Translation (using non-publicly-routable IP addressing for internal infrastructure) to protect the internal network from external harm. Few companies these days pay particular attention to securing individual development teams, the myriad internal computing resources updates, or penetration detection within a network. Fewer still actually dedicate IT resources to network security.
Im not saying that this is criminal or that I think that people who have this problem are idiots, I am simply saying that it IS a problem which is often not seen until it bites people in the delicate rear such as we see here. I mentioned a minute ago that the IT professionals which do the actual day to day work may not be entirely to blame. Most game development firms dedicate a very small expenditure to IT Departments because the IT department is probably one of the worst capital costs for a firm from which it is guarenteed there will be little or no revenue stream unless the company plans on offering application hosting to business partners or investing in a similar service not central to the corporate product line. This lower level of resource spending means that some places have to be cut short. It may mean 1 security analyst instead of 2 or 3. It may mean that the Network Administrator or Internetworking professional has to take over non-dedicated security duties in addition to primary function. This tends to compromise company security. Further, software nowadays is so difficult to patch and keep current particularly accross many desktops that you can scan a particular network and find at least a 50% difference accross your enterprise infrastructure in versioning of desktops, productivity software, and internet related applications. The aforementioned lack of resources means that companies often cant dedicated IT support time to updating machines accross the enterprise or deploying a form of Operating System/Application version management.
So, we end up with situations like the one that Half Life has been hurt by. An outdated application (Microsoft Outlook) was installed on a sensitive machine (apparently Gabe's) which had direct access to sensitive company information (the Half Life 2 source tree). There was no internal security on the source tree that we know of. There was no internal security on the machine that we are aware of. There was no segregation or sequestering of Developing network segments. And finally, it is apparent there was no version control software for keeping sensitive machines up to date. These multiple faults all contributed to the success of the recent attack and the compromise of sensitive information.
I, for one, hope this not only serves to make Valve aware of the need for information technology security, but between this attack, the recent Email worms, and blaster, more enterprises and particularly game developers take heed of recent events.
How do I think this could have been prevented? Implement multilayered security. Try to do some security forecasting for major projects. Allow a larger margin to be spent on IT resources, it pays off in the long run in avoiding events like this even if no revenue from these expeditures are showing on the balance sheet. Isolate development partially.
Ok, first, the external security. Use enterprise-level network address translation to segregate your internal infrastructure from the internet. Use protocol based proxying. Put high profile externally accessible machines on a DMZ. Use active intrusion detection (cisco makes good software for this, checkpoint used to be pretty good but are now falling slightly behind in terms of implemented technology.)
Preplan your network model. With NAT, you can build a network structure you need. Spend the extra money on layer 3 switches and build a flexible network which assigns entire ranges of class C addresses to individual company functions. Remember that using NAT with the 192.168 range allows you to use 2^16 addresses since the range is Class B. You have plenty of address space at little to no cost, so spread it around. Management should have thier own subnet. Accounting/finance should have thier own subnet. Each Development team should have thier own subnet. Common Corporate infrastructure should be on its own "server" subnet. This allows more flexibility with routing and securing access between corporate areas. Yes, it takes a little more to set up, and some foresight, but it makes events like this more difficult by segregating access points. Compromise a machine in finance and there shouldnt be any danger to development teams or development servers.
Many people in the past week have been running around "why werent the developers on a LAN"? Well, most game companies are not willing to accept the loss of productivity and the annoyance that complete segregation entails. What could have been done (since theoretically our developers are on thier own network segment, RIGHT?) is implement a segment-specific proxying and filtering. Certain common network assets are accessible, very little else can get in or out inside the company. No one that is not DIRECTLY related to that project's development has the ability to penetrate the network isolation. However developers still have internet access, can still email, and still have a reasonable level of security.
Implement product version management. There are online services to assist with this as well as stand-alone internal network products which work as a client-server system, downloading updates to a central server and then deploying them internally to company machines. Something as simple as this could have saved all this time and trouble. When gabe connected into the network, the version management sensed the network connection, requested the latest update check from the central server, and just like that outlook security fixes are patched and gabe is asked to please reboot.
There is no silver bullet, no perfect security solution short of an Adaptive Packet Destruction Filter (a pair of wire cutters for those of you not familiar with internetworking). People who are absolutely determined will eventually penetrate something. Its the job of IT and IT management to limit that exposure and make the network penetration as difficult as possible with as little impact as the IT department can possibly get away with.
This wont affect game sales but it definitely cost Valve some reputation, a huge dose of embarassment, and put thier shader routines into the wild. Valve failed this time. Dont fire your IT grunts. Hire a couple new ones. Buy some version management software, plan for an infrastructure rebuild after this game releases. Make sure it doesnt happen again.
