The Hard Realities of IT Outsourcing

The Hard Realities of IT Outsourcing

February 27th, 2007
By Stephen Northcutt
Version 1.1

Summary: Outsourcing is driven by five principal concerns: to lower cost, increase speed of growth, focus on core competency, stay compliant with government regulations, and compensate for the difficulty of recruiting and maintaining specialized hot skills talent in a world of increasing IT compensation.

Outsourcing is such a cyclical thing: one year it's all the fad, and the next year, people start pulling work back in-house. At the same time, however, the more we become globally connected, the more the pressure to outsource grows in strength. In addition, each major technical evolution creates an opportunity for offshoring and outsourcing to increase. As we go from one primary programming language, operating system, or network federated architecture to its successor, organizations realize they have an expensive workforce that lacks the new hot skills in information technology and information security. What level of IT compensation is appropriate, and what criteria determine that level? Any lower cost labor market with the ability to catch dominant new trends and educate a work force in them can then offer the needed hot skills services. While this can be disruptive, 80% of executives surveyed at a Gartner outsourcing summit[1] said they would not let a potential backlash on the part of the current workforce affect their decision to outsource. Organizations like Gartner and SANS offer meetings where expert analysts discuss coming technical trends and the management decisions leaders will have to make in response to them.

What can be outsourced? Just about everything, but common IT and IT Security tasks include:

• Code development
• Data center operations
• Log management and review
• Web page development/maintenance
• Documentation and policy
• Vulnerability Scans
• Penetration testing

Let's examine each of the five concerns that drive outsourcing: lowering cost, increasing speed of growth and reducing time to market, focusing on core competency, staying compliant with government regulations, and compensating for the difficulty of recruiting and maintaining specialized hot skills talent. . Because they are interrelated, we will cover some of these in more detail than others to avoid repetition.

Lowering cost by outsourcing and offshoring

Lowering cost is by far the biggest driver in outsourcing and offshoring: a lower labor rate is a massive advantage. Of course non-in-house labor brings its own surprises, and companies without experience often fail to achieve the IT compensation savings they hope for. Managers deliberating whether it makes economic sense to outsource one or more IT functions usually begin their analysis by looking at what the outsourcing itself will cost, and comparing that with what the company is currently spending to perform the same functions in-house. This analysis can bring up more complex factors, however, that are often overlooked. What is the real cost to your company if in-house security monitoring fails, allowing a crippling attack that an outsourced monitor, watching your network 24/7, would have prevented? Are there significant gains in productivity when outsourcing allows in-house employees to get back to focusing full-time on what they do best? What if your company's security functions are so critical that you simply can't outsource them? Cost data is important, but useful only if it's data on the real cost, and calculating that can involve consideration of variables such as these. Another factor to consider is that costs are rising so the costs you pay today may be much higher next year.[2]

Increasing the speed of growth and reducing time to market

The can be one of the most intelligent decisions for an organization to make. If your people lack either the skills or the disposition to focus on a particular aspect of the business, it might make sense to consider outsourcing. Here is a case study typical of many.

A growing business with 65 employees and annual revenue of 16M realized that a key to faster and sustained growth was marketing. Up to this point, the company's owner had written most of the advertising copy herself, but doing so took time away from her other important management responsibilities. She had received a copywriting proposal from a PR company knowledgeable in her industry, but the proposal came with a price tag of $6K - $7.5K per month. It seemed too expensive, yet to pass it up would be expensive too - quite possibly more expensive. From code testing to logistics, if your people do not have the skills and the desire to focus on a given task, outsourcing is something you should consider.

Brian Varine, author of the Outsource Monitoring section of the SANS IDS FAQ, points out three further advantages to outsourcing, as well as some disadvantages.

The first advantage applies to Managed Security Monitoring, MSM. Intrusion Detection requires a skilled person to analyze what is happening on the network. For most security administrators, however, their job description includes a variety of responsibilities in addition to being the "IDS guy". If they are lucky, they may get an hour or so of "quality" time per week with the IDS. With an outsourced MSM, in contrast, the network is monitored 24 hours a day/7 days a week. This means that at 2AM, your network is being monitored.

Second, an outsourced MSM has the personnel to figure out what is an alert and what is a false positive. With such an MSM, that 2AM call isn't going to take place unless a skilled person at the MSM thinks it's worth waking you up for.

Third, outsourced MSM gives you access to higher skill levels. With outsourcing, MSM analysts monitor your networks day and night for signs of intrusion. Over time, this gives you an enormous advantage, as these people develop the hot skills that you would otherwise have to pay a premium for. Because outsourced MSM analysts aren't limited to one network, they see attacks from a variety of sources. This allows them to recognize attack patterns much better than a person who scans logs from one network for an hour each day.

However, Brian Varine points out, there are disadvantages as well. The most obvious disadvantage is cost: MSM's are not cheap. Looking at two industry examples, Brinks Internet Security charges in excess of $8000 per month; Counterpane's monthly charge is $12,000. While this might represent a considerable cost for a large business, it is clearly not a feasible option for a small company with limited IT compensation resources. Counterpane argues that its price is competitive with having your own in-house monitoring. Looking at what a semi-skilled administrator costs, they make a valid point, especially when you consider that they provide 24/7 monitoring. Still, for most companies, a recurring monthly cost at the $12,000 level is a hard sell, especially since you still need to purchase an IDS (they monitor your IDS, but do not supply it).

Another caveat in selecting an outsourced MSM is who the company is. There are services out there that claim to be an MSM, but consist merely of a box they put on your network that sends out pages if something is detected (http://www.securityhome.com). This isn't any better than putting in your own IDS and having it send a page. The MSM you select needs to be a trusted partner. They will be the guardians of your network, and you will trust them to protect you. You can't be thinking about who is the cheapest solution; instead, think about who is the partner that you trust the most. For some organizations, no company will fit.

Inability to recruit and maintain specialized hot skills talent

We have touched on this before, but if you need specialized skills you have to be successful in both recruiting and retention. On a cross-country flight my seatmate was the HR director for Intuit on his way to a recruitment conference. "Is finding and keeping talent really that big a deal?" I asked. He said, "Imagine trying to hire programmers who are also experts in tax law." That was the moment the light went on and I really started to understand.

There is one alternative to hiring the talent you need, of course, and that is to grow it yourself. Take smart, loyal employees and get them trained. Use training from top market leaders who have instructors with actual field experience. That way, your people can start applying what they learned in class the minute they get back to the office. And don't forget assessments: there should be some sort of testing to demonstrate employee mastery of the material. This is why many organizations offer a salary premium when employees get certified, thus demonstrating they have mastered hot technical skills. One source of hot skills training for IT Security, Audit and Operations with assessment is the SANS, Stay Sharp, STAR and GIAC family of courses and assessments:

http://www.sans.org/training/courses.php
http://www.sans.org/staysharp/about.php
http://www.giac.org

Regulations are driving organizations to outsource

One effect of regulatory guidance is that people look to Managed Security Service Providers (MSSPs) for help in meeting the increasingly burdensome interpretations of government regulation. With the widespread high-profile losses of customer and employee privacy information, this trend can only increase.

According to Bruce Schneier, CTO Counterpane, compliance requires organizations to actively monitor the security of their networks in realtime, and maintain a robust audit trail of events that can be used to investigate an intrusion after the fact. Organizations simply don't have the manpower or expertise to do this kind of sophisticated monitoring in-house; outsourcing to a company like Counterpane is the only cost-effective way to maintain compliance with these regulations.

Kevin Behr, CTO of IP Services, points out that many executives mistakenly believe that when they outsource the management of their infrastructure, they are also outsourcing their regulatory obligations. The hard truth is that management, not the outsourcer, is responsible for compliance. Outsourcing is not a license to abdicate management's responsibility for compliance with all applicable state, federal and international laws.

Sidebar: Outsourcing has it's own risks

When California Senator Liz Figueroa was authoring Senate Bill 1451, a measure to ensure that medical privacy is protected under outsourcing, her fact sheet related the following story. "In October of 2003, the San Francisco Chronicle first reported the story of a medical transcriptionist in Pakistan who had subcontracted with a Florida firm that had a contract with a Texas company that did business with the Medical Center at UC San Francisco. The woman claimed she was not being paid for her work and threatened to publicly release some of the information she possessed if she was not paid immediately. The UCSF Medical Center was not even aware this information had left the country, though it appears no laws had been broken until the Pakistani transcriptionist threatened to reveal the highly confidential medical information. While the woman was eventually paid and no records were actually released, the incident exposed a dangerous problem."

http://democrats.sen.ca.gov/senator/figueroa/

According to Susan Orr, Vice President of Regulatory Compliance for Catbird Networks, the following specifics should be evaluated when considering outsourcing IT security tasks to achieve regulatory compliance:

• Review of third party audits
• Review of tests/test results performed on service provider's controls
• Review results of security configuration tests such as Center for Internet Security metrics on the service provider's servers and routers
• Review of vulnerability assessments, how they are performed, how often, remediation, what is the patch management process following identification of a vulnerability
• Performance monitoring
• Service Level Agreement compliance
• Antivirus protection
• Firewall port scans
• Website monitoring including defacements and web linking
• Rogue LAN and wireless devices
• Evidence IDS/IPS are used effectively

The medical transcriptionist story exposes a significant problem with outsourcing. What if your service provider is also outsourcing? Contracts may not be enough to achieve oversight, but they can be used to your organization's advantage. The contract is your opportunity to specify your security requirements for your service provider. Elements to consider include:

• Every employee of the service provider with any possibility of access to your data signs a confidentiality agreement, with severe penalties for intentional or accidental disclosure.
• The provider agrees to the strict principle of least privilege and separation of duties, and any access is granted only with the need to know.
• The provider agrees to notify you of any outsourcing to additional providers they currently use, and to notify your organization of any outsourcing they begin to use in the future. If your organization does not approve the new service provider, that should be grounds for terminating the contract.

The most important thing to consider is outsourcing and offshoring does not eliminate risk, it changes risk. New risk is introduced and threat models change, certainly the threat surface changes[3]. This was well illustrated by the fiber optic outage in China December 28, 2006 and took over two months to repair. That must have seriously interrupted some outsourcing/offshoring operations[4]. While there are certainly risk issues and intellectual property may be lost or compromised some information warfare claims may be overblown. In 2003 predictions of major problems by 2006 do not appear to have been realized[5]. Of course what we do not know . . .[6]

Focus on core competency

This may be the best reason of all to outsource. No organization can be the best in the world at everything. A good management practice is to review all business operations and units every year. If a part of the business is underperforming or has a high error rate, that part is certainly a candidate for remediation, and it might also be a candidate to outsource or divest.

What is the right decision for your organization? Globalization is an increasingly dominant trend and you should be considering outsourcing anything you know you can never be the best in the world at. So, it really is not a question of if, but when. Set up a tickler calendar entry at least every six months to reconsider the question. Start with something that isn't too critical to your operations so you can build your outsourcing skills at minimum risk. And know why you are deciding to outsource, in this article we have shared five logical reasons: to lower cost, increase speed of growth, focus on core competency, stay compliant with government regulations, and compensate for the difficulty of recruiting and maintaining specialized hot skills talent in a world of increasing IT compensation.

[1] http://www.gartner.com/DisplayDocument?doc_cd=116614
[2] http://money.cnn.com/2004/07/13/technology/techinvestor/lamonica/
[3] http://infosysblogs.com/managing-offshore-it/2006/11/yes_james_offshoring_is_fraugh.html
[4] http://www.app.com.pk/en/index2.php?option=com_content&do_pdf=1&id=1201
[5] http://www.computerworld.com/careertopics/careers/story/0,10801,80935,00.html
[6] http://www.computerworld.com/managementtopics/outsourcing/story/0,10801,84723,00.html

NOTE: some of this information is based on an article by Stephen Northcutt for Cyber Defense Magazine 2004