2009 October

Selecting Your Data Center Part 2 – Geography and Location

On October 30, 2009, In Energy, Internet and Telecom, Virtualization, cloud computing, By Administrator

Data center selection is an exercise in compromise. Everybody would like to have the best of all worlds, with a highly connected facility offering 24×7 smart Selecting the Data Center Location hands support, impenetrable security, protection from all natural and man-made disasters, in addition to service level agreements offering 5-Nines power availability at $.03/kW. Not likely we will be able to hit all those desired features in any single facility.

Data center operators price their facilities and colocation based on several factors:

Cost of real estate in their market
Cost of power and utilities in their market
Competition in their market
Level of service offered (including power, interconnections, etc)
Quality of facility (security, power density, infrastructure, etc)

Networks, Content Providers, Enterprises, and Eyeballs

The basic idea of an Internet-enabled world is that eyeballs (human beings) need to access content, content needs access to eyeballs, eyeballs and content need access to networks (yes, eyeballs do need to communicate directly with other eyeballs), and networks need access to content and eyeballs. Take one of the above out of the equation, and the Internet is less effective. We can also logically add applications to the above model, as applications are now communicating directly with applications, allowing us to swap eyeballs for apps to complete the high level model.

Organizations using the Internet fall into a category of either a person, an application (including enterprise, content, and entertainment applications), or a network (including access, regional, and global networks).

Each potential organization considering outsourcing some or all of their operations into a data center needs to ask themselves a few basic questions:

Is the organization heavily dependent on massive storage requirements?
Is the organization highly transaction-oriented? (such as a high volume eCommerce site)
Is the organization a content delivery network/CDN, requiring high bandwidth access to eyeballs?
Are your target applications or eyeballs local, regional, global?
Is the company a network service provider highly dependent on network interconnections?

Storage and servers = high density power requirements. The more servers, the higher the operational expenses on both space and power. This would logically drive a potential collocation customer to a location with the cheapest power – Data Center Elements however that might be a location outside of central business districts, and possibly outside of an area well connected with domestic and international telecom carriers, network service providers, and access networks (including the cable TV networks serving individual subscribers).

Thus the cost of power and real estate might be favorable if you are located in Iowa, however bringing your content to the rest of the world may limit you to one or two network providers, which with limited competition will likely raise the price of bandwidth.

Locating your business in a city center such as New York or Los Angeles will give you great access to bandwidth through either a colocated carrier hotel or carrier hotel proximity. However, the cost of real estate and power in the city center will be a multiple of that you may find in areas like Oregon or Washington State.

In a perfect telecom world, all networks and customers would have access to dark fiber from facility-based carriers serving the location they are either located or doing business. Allied Fiber’s Hunter Newby believes that facility-based carriers should be in the business of providing the basic “interstate highway” of communications capacity, allowing any company who can afford the cost to acquire high capacity interconnections to bring their operation closer to the interconnection points.

If you follow the carrier world you will know that at least in the United States, carriers are reluctant to sell dark fiber resources, preferring to multiplex their fiber into “lit” circuits managed and provisioned by the carrier. Clearly that provides a lot more potential revenue than selling “wholesale” infrastructure. Also makes it a lot more expensive for a company considering collocation to locate their facility in a geography separated from the major interconnection sites.

The Business Case and Evaluation

Again, selecting your desired location or locations to outsource your business is a compromise. In the United States Virginia is a good location for power, and an expensive location for interconnecting and collocating. Los Angeles is among the lowest cost areas for interconnections, mid way up the power scale, but more expensive for space.

Consider the possibility of moving to a great location in Idaho, with low cost power, and low cost real estate. You build a 500,000sqft facility, with more than 300 watts/sqft power capability. Your first project supports more than 20,000 servers delivering Internet streaming media content. Your facility costs are low, but your network costs become very high. You cannot buy dark fiber from a facility-based carrier, and the cost of leasing 10G wavelengths is nearly $10,000/month per wavelength. You probably have 500GB of data to push into the Internet. Is the power cost vs. connectivity and bandwidth compromise in your favor?

Here is another exercise. Let’s say for argument, in a Los Angeles carrier hotel static costs may run:

$1000/month for a cabinet in the carrier hotel, $500/month for a cabinet in nearby facility.
$12/breakered amp (breakered amps are still the norm, moving to usage-based models)
$200/month for a cross connection within the carrier hotel building
$1000/month for a fiber cross connect to a nearby or adjacent building
$1000/month for an Internet Exchange Point/IXP connection (if you are a network service provider)

NOTE: Los Angeles has several large carrier hotels in the downtown area, as does New York, with buildings such as 60 Hudson and 111 W. 8^th offering potential tenants multiple options. Other cities such as Seattle, Miami, and Chicago have more limited options, with a single dominant carrier hotel.

If you are a medium sized network service provider, you may consider getting a couple cabinets in a nearby facility and acquire a couple fiber cross connections to one or more nearby carrier hotels. Get a cabinet within the carrier hotel, add high capacity switching or routing equipment in the cabinet, and then try to maximize the number of local cross connects with other networks and content providers, and connect to a local Internet Exchange Point for additional peering flexibility.

Then take your same requirement for both cabinet space and interconnections, and try the evaluation in several different cities and markets. Fit the cost into one of the above squares in the Data Center Basic Elements chart, and determine the cost for each component.

If your business requirement is more dependent on space, and that is the highest potential operational expense, then you need to consider which location will minimize cost increases in the other three quadrants while you evaluate the best location for meeting your space budget. If your requirement spans several different geographies, add the cost of interconnection between locations to your interconnection costs. Does the location give you adequate access to the target applications or eyeballs?

If you find that a location in Omaha, Nebraska, meets all your requirements, but your target audience also includes a high percentage in India or China, then the cost of getting to your eyeballs in both OPEX and performance may make the Nebraska site untenable – even though it meets your high level budget.

Enter the Cloud

Nearly all businesses and organizations now have an additional alternative. The virtualized commercial cloud service provider. Virtualization products have come a long way over the past couple years, and are maturing very quickly. CSPs such as Google, Amazon, Rackspace, and Layered technologies are providing very powerful applications support for small and medium business, and have become a very visible debate at the national level as governments and large corporations deal with questions of:

Focusing on their core competencies, rather than internal IT organizations
Building more efficiency into the IT infrastructure (heavy on energy efficiency)
Recovering space used by IT and computer rooms
Reducing OPEX spent on large IT support staff
Better technologies such as netboooks
And more…

Thus the physical data center now has competition from an unlikely source – the cloud. All new IT and content-related projects should consider cloud computing or software as a service (SaaS) models as a potential alternative to bricks and mortar data center space.

Many venture capital companies are now requiring their potential investments to consider a hosted or SaaS solution to outsource their office automation, web presence, and eCommerce applications. This is easily done through a commercial web service or cloud hosting company, with the additional option of on-demand or elastic expansion of their hosting resources. This may be the biggest potential competitor to the traditional data center. The venture community simply does not want to get stuck with stranded equipment or collocation contracts if their investment fails.

Disaster Recovery and Business Continuity

One final note on selecting your location for outsourcing. Most companies need some level of geographic diversity to fulfill a business need for offsite disaster recovery apps and storage, load balancing, proximity (to eyeballs and applications), and interconnections. Thus your planning should include some level of geographic diversity, including the cost of interconnecting facilities to mirror, parse, or back up files. The same rules apply, except that in the case of backup the urgency for high density interconnections is lower than the primary operating location.

This does raise the potential of using facilities in remote locations, or locations offering low cost collocation and power pricing for backups.

Links to Data Center Resources

Here are a couple links to magazines and eZines supporting the data center industry.

Part 3 will explore the topic of understanding the hidden world of data center tiers, mechanical and electrical infrastructure, and site structure.

John Savageau, Long Beach

Prior articles in this series:

28Oct09

Selecting Your Data Center Part 1 – Understanding the Market

Tagged with: carrier hotel • cloud computing • colocation • data center

Selecting Your Data Center Part 1 – Understanding the Market

On October 28, 2009, In Internet and Telecom, Virtualization, cloud computing, By Administrator

The data center industry continues to evolve with mergers, acquisitions, and a healthy crop of emerging companies. New data center products and services Old Data Center are hitting the street, an aggressive debate on the model of selling space vs. power, and alternatives to physical data center space in the cloud are giving us a confusing maze of alternatives to meet our outsourcing needs.

The data center market is not unique. For example, in Southern California we have a wide variety of supermarkets and grocery stores including VONs, Ralphs, Albertsons, Jons, Trader Joes, Whole Foods, and lots of others. All grocery stores basically sell the same kinds of products, with very few exceptions.

What makes you go to VONs, rather than Whole Foods? Is it location? Prices? Image? A social issue?

The data center industry is not significantly different. In a city such as Los Angeles you have Equinix, Switch and Data, Savvis, BT Infonet, CoreSite, US Colo, Digital Realty, Level 3 – just to name a few. What makes one facility more attractive than another to fulfill your collocation needs?

Data centers, at the most common denominator, have traditionally offered:

Concrete (space for cabinets, racks, cages, suites, etc)
Power
Air conditioning
Interconnections

If all data centers offer the basic components listed above, then what discriminates the data centers from one another?

Now we can add additional alternatives to the basic data center model – the public cloud services provider/CSP and Software as a Service/SaaS.

As a potential data center tenant (this includes “virtual” data center tenants living in a CSP infrastructure) we have to evaluate all the above components, and determine which collocation or data center provider will best meet our facility, budget, and connectivity needs.

The Sense of Urgency

The CIO of the United States, Vivek Kundra, recently pressed the case for data center consolidation within the US government, as well as offering a strong recommendation that the US data industry strongly consider moving their operations into either consolidated data centers or virtualize within a cloud provider.

It is clear that data centers used by small and medium companies, as well as most content delivery companies, find better efficiencies in bringing their eCommerce and Internet-facing parts of their business into the data center, and locally interconnect with the Internet service provider community.

The cost of building a data center, providing staffing to manage the data center, and ensuring the efficiency of power and cooling usage is beyond the core competence of most companies. The need for disaster recovery plans, offsite storage, and other business continuity planning are just a few of the long list of items we need to consider as part of an overall information technology/IT or general business plan.

The potential waste of operational expenses, capital budgets, and resulting market “opportunity cost” justifies all companies at least consider outsourcing all or some of their IT operations – particularly as data center and CSPs increase their capabilities.

With the availability of netbooks, online applications (SaaS), and server-based office automation products, all companies should put this on their annual review list. Even the Los Angeles Police Department (LAPD) recently announced their decision to outsource the email to Google. This model does not appear to be going away anytime soon.

The “Selecting Your Data Center” Series

This series will walk through the process of identifying the need for outsourcing, identifying the best location for your data center, discriminating between the alternatives, and finally getting to your decision.

We welcome all comments, experiences, and discussions related to the data center community that would provide productive feedback for a potential data center or CSP tenant.

John Savageau, Long Beach

Tagged with: cloud computing • data center • ecommerce • interconnection • outsourcing • saas • slecting data center

Shrouding the Net Neutrality Debate in a Cloud of Politics

On October 26, 2009, In Government Regulatory, Internet and Telecom, By Administrator

The FCC finally moved the network neutrality debate forward Thursday, voting to begin developing open Internet regulations. The topic has become quite interesting over the past week, as strong-willed proponents and opponents of Internet Network Neutrality net neutrality turn up campaigns to influence law makers prior to voting on any net neutrality principles that may become law.

The debate is actually quite simple – should the government regulate, or not regulate the Internet? That discussion revolves around the six principles of network neutrality proposed by the FCC:

Under the draft proposed rules, subject to reasonable network management, a provider of broadband Internet access service:

would not be allowed to prevent any of its users from sending or receiving the lawful content of the user’s choice over the Internet;
would not be allowed to prevent any of its users from running the lawful applications or using the lawful services of the user’s choice;
would not be allowed to prevent any of its users from connecting to and using on its network the user’s choice of lawful devices that do not harm the network;
would not be allowed to deprive any of its users of the user’s entitlement to competition among network providers, application providers, service providers, and content providers;
would be required to treat lawful content, applications, and services in a nondiscriminatory manner; and
would be required to disclose such information concerning network management and other practices as is reasonably required for users and content, application, and service providers to enjoy the protections specified in this rulemaking.

The Pro Argument of Network Neutrality

Oddly, former adversaries Google and Verizon issued a joint statement regarding their position on net neutrality. Both companies have significantly changed their positions since the debate originally hit the headlines in 2005, with a highlight of the joint statement:

“For starters we both think it’s essential that the Internet remains an unrestricted and open platform — where people can access any content (so long as it’s legal), as well as the services and applications of their choice.
Transformative is an over-used word, especially in the tech sector. But the Internet has genuinely changed the world. Consumers of all stripes can decide which services they want to use and the companies they trust to provide them…

…This kind of “innovation without permission” has changed the way we do business forever, fueling unprecedented collaboration, creativity and opportunity. And because America has been at the forefront of most of these changes, we have disproportionately benefited in terms of economic growth and job creation.”

This oddly puts Verizon in opposition to the other main anti-net neutrality supporters such as AT&T, Cox Communications, and Comcast.

Certainly companies like Google have come a very long way from 2005 when the debate was clearly one of who pays whom, for what kind of service, and who has the right to determine the quality of services over basic Internet infrastructure. In the early days content providers wanted a level of government regulation to ensure the Internet transmission and network providers did not have control over the end user experience, and objected to statements by AT&T’s then CEO Ed Whitacre who stated in an interview with Business Week:

“How do you think they’re going to get to customers? Through a broadband pipe. Cable companies have them. We have them. Now what they would like to do is use my pipes free, but I ain’t going to let them do that because we have spent this capital and we have to have a return on it. So there’s going to have to be some mechanism for these people who use these pipes to pay for the portion they’re using. Why should they be allowed to use my pipes?

The Internet can’t be free in that sense, because we and the cable companies have made an investment and for a Google or Yahoo or Vonage or anybody to expect to use these pipes [for] free is nuts!

This set off both a furor among Internet users, as well as a new movement to ensure the “telcos” did not ever again have an opportunity to restrict or limit free access to their networks. Arguments ranged from identifying the US taxpayer and AT&T customers paying for basic infrastructure our of Universal Services Fund/USF (meaning the US taxpayer is actually the owner of much of AT&T’s USF-funded infrastructure), to AT&T subscribers being forced to use AT&T preferred content providers based on their control of the network.

There are many organizations representing both users and Internet industry companies supporting the idea of network neutrality. The current law in the house is sponsored by Reps. Edward Markey (D-Mass.) and Anna Eshoo (D-Calif.) who introduced the Internet Freedom Preservation Act of 2009 (H.R. 3458) in July 2009.

The Con Argument

Not surprisingly, the Con argument is dominated by conservatives in both the government and corporate communities.

John McCain (who is also accused by the Huffington Post as having received more than $800,000 in campaign funding by AT&T, Verizon, and Comcast) rejected the FCC’s vote, and offered a new proposal called the “Internet Freedom Act.”

“Today I’m pleased to introduce the Internet Freedom Act of 2009 that will keep the Internet free from government control and regulation,” McCain said in a statement. “It will allow for continued innovation that will in turn create more high-paying jobs for the millions of Americans who are out of work or seeking new employment. Keeping businesses free from oppressive regulations is the best stimulus for the current economy.” (CNN)

The only thing missing in the above cliché-filled statement is a series of pictures of crying babies, unemployment lines, and California wildfires. The bottom line here is that politicians, bending to pressure or contributions from opposition parties, will use any words available to tug at either emotions or heart strings, regardless of the presence of factual data to support the position.

AT&T allegedly sent notice to all their employees, including union members and families, to write their representatives in favor of knocking down network neutrality. Perhaps that is a natural activity in the political process, however bandwagon appeals without supporting fact will not give the American people the broadband environment needed to compete in the global market placed.

Let’s look at both arguments in detail. Do a Google, Bing, Yahoo, or other web search on the topic of Net/Network Neutrality. You will find a lot of web references, news stories, blogs, and opinions on the topic. Much of it anarchistic noise, much of it very valuable information.

Hunter Newby, CEO of Allied Fiber, asks the question “What if the United States falls further behind Europe in deployment of broadband networks? What if we lose track of the need to wire each and every community? What if the United States falls so far behind Europe and the rest of the world due to politics preventing innovation that we can never economically recover?”

There are those who still believe the carriers, such as Verizon, AT&T, broadband wireless providers (such as Clearwire), and the cable companies should concentrate their efforts on delivering connectivity to each and every addressable community in the United States. Facility-based carriers (those who own the physical cable) should concentrate on providing bundles of “big, fat, dumb, communications pipes.”

Comments on Net Neutrality are Now Open with the FCC

The great thing about the US system is that no national law is ever a unilateral decision. We have a wonderful system of due-diligence through the congress and senate, with support from the executive and judicial branches of government.

The Federal Communications Commission under the leadership of Chairman Genachowski has opened discussion and the period of comment on the FCC guidelines. The period for comments is open until 14 Jan 2010. Some links for those interested in the topic include:

The question arises, “what if we, as Americans interested in the future of the Internet, American innovation, the American economy, and our future generations actually took the time to read through the issue of Network Neutrality? What if we used our research as a basis for making our own decision on which side of the debate we fall, or if there is yet another strong argument to consider?”

It is a difficult topic, with a lot of noise and clouds shrouding the core issues. Weigh-in, let us know your opinion.

John Savageau, Long Beach

Tagged with: fcc • genachowski • google • h.r. 3458 • hunter newby • john mccain • markey • net neutrality • network neutrality • verizon

Managing Emergency Communications – A Tutorial with Marc Ladin

On October 23, 2009, In Internet and Telecom, LA Life, By Administrator

The Station Fire ripped through communities along the northern rim of Los Angeles in August and September, consuming an area more than 160,000 acres. Evacuations came with little or no warning, homes and buildings lost, and the entire ordeal put a tremendous strain on utilities and resources. Including water.

When the city of Glendale needed to quickly alert residents to lower their water and power use to enable fire fighters to gain access to critical resources, they turned to a local company, Everbridge, to reach citizens with real-time notifications alerting them to the emergency.

On Thursday night Marc Ladin, VP of Global Marketing at Everbridge, walked CTC members though an introduction to emergency and incident communications management.

The Need for Emergency Management

Communications technology has made incredible leaps in utility, applications, ands capacity over the past few years. We can reach nearly any point or person in the world through telephone, mobile phones, Internet email, Twitter, Blackberry messaging, radio, television – the list is becoming endless.

Regardless of the technologies, natural and man-made disasters and problems remain a part of our lives, and will always be part of our lives. Our businesses, governments, and even survival, depends on how we prepare for disaster, and are able to respond to events that touch our lives. Good events and bad.

Marc Ladin makes a living solving the problem of communicating during emergencies and events. The residents of Glendale, like most communities in the United States, offers residents the option of registering their preferred communications devices with the city.

This gives the city an immediate channel to reach and inform residents in the event of disasters and other incidents of interest or impact to the city and residents.

In the case of the Station Fire, Glendale was able to immediately reach enough residents, and the city was able to lower residential utility draw to the level fire fighters had adequate water resources to protect the community.

The same model applies across the spectrum of emergency notification.

The Enterprise Business Continuity Plan

Nobody wants to think of a disaster that will hurt people, or isolate them from their family or organizations. However, it is also clear that any organization needs to have a business continuity plan in place, and a disaster response plan in place to allow the organization to quickly respond to, and manage, any event that will potentially damage the organization’s ability to function.

Consider this scenario. A large multi-national chemical products company. Highly visible in the world business community, and customers located around the world.

The worst case scenario happens. At the HQ site an explosion occurs in the manufacturing plant, killing several person in senior leadership roles, and requiring a massive response by emergency services and evacuation in the surrounding community.

Who do we need to notify to respond to the emergency, and who needs to know about the problem?

First responders – fire fighters, HAZMAT teams, ambulances, local hospitals, police
Local Community – residents, media (radio and television)
Company leadership – management, public affairs, operations
National and global media

How do you get the message – the real message – out to those people?

How do we determine if somebody is trapped in the disaster area, and needs help?

The process is getting easier. Every person, machine, and device connected to the Internet or other global communications service can be part of the event notification process.

Registering Your Communications Device for Notification

A company such as Everbridge offers as utility for managing emergency and event notifications. The utility (Everbridge) operates as a SaaS (Software as a Service) application, physically separated from the users. The SaaS application resides on several geographically diverse data centers, with multiple communication providers providing the conduit for global device notification access.

An organization will compile a table of their users and devices, with an individual having the ability to register all their available communications devices (mobile phones, email, Twitter accounts, etc), including a preference on notification priority (i.e., mobile phone message first, email second, home phone third…).

The organization then has the ability to sort members into different categories of notification. An example of how an organization might be sorted is:

C-level management notifications
Persons notified during emergencies
Geography (everybody in the Long Beach office, everybody in the Atlanta office, everybody in Japan, etc)
Function (operations, engineering, marketing and sales)
Local area first responders
And any other desired sort

Of course a single entry is easily tagged for multiple notification categories.

How to Make a Notification

In a traditional environment company leadership wants to make a notification. They may have their secretary make phone calls, might call an operations center and open a notification checklist, or other time-tested process.

The modern notification system can use a wider variety of methods for generating a notification:

A human being opens a web page and types in a notification message for distribution
A human being prepares an email or SMS message, and sends it to an address that spawns the desired notification tree
A machine experiences a condition that requires a human response
- Fire alarm
- Equipment failure
- Security break-in or event
- Etc

Once the message is triggered, and the notifications made, then you need to make a decision on whether or not the notified persons need to acknowledge or respond to the notification. Modern systems also manage and automate the acknowledgement process by logging replies to the notification message, allowing the alert initiator to determine if everybody has received the message.

This is important if you are managing a disaster, and need to determine if somebody could potentially be hurt or in danger, or if you need to escalate a decision situation to the next person in a business continuity plan.

With GPS capability, it is now even possible to determine the exact location of a desired device, further helping locate persons in a disaster. Consider a heart patient with an active monitoring device – that device can be registered in a hospital, first-responder, family, and neighbor notification matrix. This will increase the probability that person will survive in the event of health problems.

Other Creative Ways to Use a Notification System

Of course the same system that handles emergencies can also handle positive messages. The marketing group can use the same notification system for press releases, management can deliver positive company results to employees – basically once the person and device/s are registered in a data base, the entry can be used for whatever desired.

Marc Ladin presented a great vision. His company is putting the vision into reality, and has a lot of exciting features available today, and in the mill for tomorrow.

John Savageau, Long Beach

Tagged with: email • emergency notification • everbridge • marc ladin • messaging • mobile phone • twitter

Deleting Your Hard Drives – Entering a Green Data Center Future of SSDs

On October 20, 2009, In Energy, Internet and Telecom, environment, By Administrator

For those of us old-timers who muscled 9-track tapes on 10 ft tall on Burroughs B-3500 mainframe computers tape drives, with a total storage capacity of about 5 kilobytes, the idea of sticking a 64 gigabyte SD memory chip into my laptop computer is pretty cosmic.

Disk Drive Terms like PCAM (punch card adding machines) are no longer part of the taxonomy of information technology, nor would any young person in the industry comprehend the idea of a disk platter or disk pack.

Skipping a bit ahead, we find a time when you could purchase an IBM “XT” computer with an integrated 10 megabyte hard drive. No more reliance on 5.25″ or later 3.5″ floppy disks. Hard drives evolved to the point “Fryes” will pitch you a USB or home network 1 terabyte drive for about $100.

Enter the SSD

October 2009 brings us to the point hard drives are now becoming a compromise solution. The SSD (Solid State Disk) has jumped on the data center stage. With MySpace’s announcement they are replacing all 1770 of their existing disk drive-based server systems with higher capacity SSDs, and quoted that SSDs use only 1% of the power required by disk drives, data center rules are set to change again.

SSDs are efficient. If you read press releases and marketing material supporting SSD sales you will hear numbers like:

“…single-server performance levels with 1.5GB/sec. throughput and almost 200,000 IOPS
… a 320GB ioDrive can fill a 10Gbit/sec. Ethernet pipe
… four ioDrive Duos in a single server can scale linearly, which provides up to 6GB/sec. of read bandwidth and more than 500,000 read IOPS (Fusion.io)

This means not only are you saving power per server, you are also able to pack a multiple of existing storage capacity into the same space as currently possible with traditional disk systems. As clusters of SSDs become possible through additional tech development of parallel systems, we need to mentally get our heads around the concept of a three dimensional storage system, rather than a linear systems used today.

The concept of RAID and tape backup systems may also become obsolete, as SSDs hold their images when primary power is removed.

Now companies like MySpace will be in a really great position to re-negotiate their data center and colocation deals, as their actual energy and space requirements will potentially be a fraction of existing installations. Even considering their growth potential, the reduction in actual power and space will no doubt give them more leverage to use in the data center agreements.

Why? Data center operators are now planning their unit costs and revenues based on power sales and consumption. If a company like MySpace is able to reduce their power draw by 30% or more, this represents a potentially huge opportunity cost to the data center in space and power sales. Advantage goes to the tenant.

The Economics of SSDs

Today, the cost of SSDs is slightly higher than traditional disk systems. Even with fiber channel or Infiniband supporting large disk (SAN or NAS) installations. According to Yahoo Tech the cost of an SSD is about 4 times that of a traditional disk. However they also indicate that cost is quickly dropping, and we will probably see near parity within the next 3~4 years.

Now, if we remember the claim MySpace made that with the SSD migration they will consume only 1% of the power used by traditional disk (that is only the disk, not the entire chassis or server enclosure). If you look through a great white paper (actually it is called a “Green Paper”) provided by Fusion.io you will see that implementation of their SSD systems in a large disk farm of 250 servers (components include main memory, 4xnet cache, 4x tier 1/2/3 storage, tape storage) you will see a reduction from 146.6kw to 32kw for the site.

Data centers can charge anywhere from $120~$225/kw, showing that we could potentially, if you believe the marketing material, see a savings of $20,000/month @ $180/kw. This would also represent 47 tons of carbon, using the Carbon Footprint Calculator.

Fusion .io reminds us that

“In 2006, U.S. data centers consumed an estimated 61 billion kilowatt-hours (kWh) of energy, which accounted for about 1.5% of the total electricity consumed in the U.S. that year, up from 1.2% in 2005. The total cost of that energy consumption was $4.5 billion, which is more than the electricity consumed by all color televisions in the country and is equivalent to the electricity consumption of about 5.8 million average U.S. households.

• Data centers’ cooling infrastructure accounts for about half of that electricity consumption.

• If current trends continue, by 2011, data centers will consume 100 billion kWh of energy, at a total annual cost of $7.4 billion and would necessitate the construction of 10 additional power plants. (from “Taming the Power Hungry Data Center”)”

When we consider the potential impact of data center consolidation through use of virtualization and cloud computing, and the rapid advancements of SSD technologies and capacities, we may be able to make a huge positive impact by reducing the load Internet, entertainment, content delivery, and enterprise systems will have on our use of electricity – and subsequent impact on the environment.

Of course we need to keep our eyes on the byproducts of technology (e-Waste), and ensure making improvements in one area does not create a nightmare in another part of our environment.

Some Additional Resources

StorageSearch.Com has a great listing of current announcements and articles both following and describing the language of the SSD technology and industry. There is still a fair amount of discussion on the quality and future direction of SSDs, however the future does look very exciting and positive.

For those of us who can still read the Hollerith coding on punch cards, the idea of >1.25TB on and SSD is abstract. But abstract in a fun, exciting way.

How do you feel about the demise of disk? Too soon to consider? Ready to install?

John Savageau, Long Beach

Tagged with: burroughs b3500 • data center • disk system • environment • green • myspace • ssd

Recent and Fun California Green Energy Initiatives

On October 18, 2009, In Burbank, Energy, LA Life, environment, By Administrator

“If everyone purchasing a room air conditioner in 2009 chooses an ENERGY STAR qualified model, it would save 390 million kilowatt-hours of electricity a year. That would prevent more than 600 million pounds of greenhouse gas emissions each year—equivalent to taking more than 50,000 cars off the road—and save consumers over $43 million each year in energy bills.” (Pickens Plan Fact of the Day, 8 Oct 09)

California has always prided itself as being a leader in alternative energy innovation. Driving through the hills around Livermore, Palm Springs, or between San Diego and Yuma bring skylines full of wind turbines. The California Energy Commission claims that wind Solar Panels for California turbines generated 6,802 gigawatt-hours of electricity – about 2.3 percent of the state’s gross system power. By the end of 2009 California actually expects to hit nearly 5% energy production from renewable sources.

While the wind turbine program has slowed down a bit due to animal rights groups objecting to bird casualties due to propeller strikes, California has not slowed down at all in the state’s aggressive goals for green energy production. While it is probably a bit too aggressive, California’s Energy Commission has set a goal of hitting 20% by the end of 2010 (Senate Bill 107), and 33% by the end of 2020 (Executive Order S-14-08).

The US Congress is shooting for 20% renewable energy production nationwide by 2010 – a far lower threshold than desired in California.

Energy Programs and Incentives in California

Each state has some level of renewable energy initiative supporting energy efficient homes. California’s program falls under the “The California Energy Commission’s New Solar Homes Partnership” (NSHP). This is a great resource not only for existing home owners in California, but also those persons planning to build new structures. The objectives of NSHP’s program include:

In the Home

A solar home with high energy-efficiency features offers homeowners:

Clean, renewable energy
Utility bill savings
Predictable utility costs
Protection against future rising electricity costs

California is also offering financial incentives to homebuilders to design energy efficiency and the potential of renewable energy planning into the new home. Solar energy “is one of the most significant personal actions one can take to cut air pollution and greenhouse gas emissions, while helping to conserve precious energy resources for future generations. Plus, it reduces the need for costly new power plants” according to the NSHP.

All California homeowners implementing solar panels in their homes also qualify for the federal tax credits up to $2000.

An unscientific jog around the Sunset Canyon Drive area of Burbank on 17 Oct 2009 tallied around 1 of every 5 homes observed supporting some level of solar panel on the property, visible from street level. Using guidelines from the National Renewable Energy Labs (NREL) you will see the average family in the Los Angeles area will save nearly $714 a year with solar panels supplementing their electrical supply.

For us apartment and condo-dwellers, that could almost pay 100% of our energy requirement during normal conditions, if we have a means of storing energy during evening hours and period of bad weather.

Don’t forget our earlier discussions on other simple things such as painting your rooftop white, or using solar reflective material on your roof to reduce the amount of heat in your home during the summer. By the way… you also get a one-time energy credit for that simple task.

“More than 50% of the energy used in a typical American home is for space heating and cooling. Much of that conditioned air escapes through poorly sealed, under-insulated attics. Only 20% of homes built before 1980 are well insulated. Properly sealing and insulating your attic can save you up to 10% annually on energy bills.” (Pickens Plan Fact of the Day7 Oct 2009)

In Commercial Sites

Companies such as the Bank of America (in Riverside, California) have built their facility with solar covering the entire rooftop of the building. Not only do they enjoy a tremendous savings in energy costs, but with a commercial property the BoA will receive a 30% federal construction tax credit, accelerated equipment depreciation, and additional financing support to help defray the cost of installing renewable energy resources.

California will tack on an additional incentive of $1.90 per watt up to a 1Megawatt solar panel system.

All focused on getting us to that 20% milestone in 2010, and the world-leading 33% renewable energy target for 2020.

Some Resources to Look at During Energy Awareness Month (October)

The State of California, California’s energy utilities, and the US Department of Energy have great resources to guide us in meeting our energy awareness and energy planning goals. Here is a partial list, but a great start. The Internet and Google searches will help lead you further.

The California Energy Commission Home
California Renewable Energy Handbook
Go Solar California Home
California Solar Initiative
SoCal Edison solar initiative website
PG&E solar initiative website
State of California CSI rebate calculator
US Department of Energy Solar Initiatives
The Pickens Plan

What Are You Doing?

Share your energy stories with us. What has worked for you? What has failed? Are you an alternative and reneable energy skeptic like Texas’ Governor Rick Perry? Are you an energy leader? Let us know.

John Savageau, Long Beach

Tagged with: california • california energy commission • green • renewable energy • solar energy

How Green is Your Data Center?

On October 17, 2009, In Energy, Internet and Telecom, environment, By Administrator

Data Center “X” just announced a 2 MegaWatt expansion to their facility in Northern California. A major increase in data center capacity, and a source of great joy for the company. And the source of potentially 714 additional tons of carbon introduced each month into the environment.

Think Green and Efficient Many groups and organizations are gathering to address the need to bring our data centers under control. Some are focused on providing marketing value for their members, most others appear genuinely concerned with the amount of power being consumed within data centers, the amount of carbon being produced by data centers, and the potential for using alternative or clean energy initiatives within data centers. There are stories around which claim the data center industry is actually using up to 5% of power consumed within the United States, which if true, makes this a really important discussion.

If you do a “Bing” search won the topic of “green data center,” you will find around 144 million results. Three times as many as a “paris hilton” search. That makes it a fairly saturated topic, indicating a heck of a lot of interest. The first page of the Bing search gives you a mixture of commercial companies, blogs, and “ezines” covering the topic – as well as an organization or two. Some highlights include:

42U (Private Company Pitching Green)
Green Data Center Blog
Web Host Industry Review – Green Data Center Info (ezine)
http://www.greenhousedata.com/ (private advocacy group)
http://www.thegreengrid.org/ (industry Organization)

With this level of interest you might expect just about everybody in the data center industry to be aggressively implementing “green data center best practices.” Well, not really. In the past month the author (me!) toured not less than six commercial data centers. In every data center I saw major best practices violations, including:

Large spacing within cabinets forcing hot air recirculation (not using blanking panels, as well as loose PCs and tower servers placed adhoc within a cabinet shelf)
Failure to use Hot/Cold aisle separation
High density cabinets using open 4 post racks
Spacing in high density server areas between cabinets
Failure to use any level of hot or cold air containment in high density data center spaces, including those with raised floors and drop-ceilings which would support hot air plenums

And other more complicated issues such as not integrating the electrical and environmental data into a building management system.

The Result of Poor Data Center Management

The Uptime Institute developed a metric called Power Utilization Efficiency (PUE) to measure the effectiveness of power usage within a data center. The equation is very simple, the PUE is the total facility powe3r consumption divided by the amount of power actually consumed by either internal IT equipment, or in the case of a public data center customer-facing or revenue-producing energy consumed. A factor of 2.0 would indicate for every watt consumed by IT equipment, another watt is required by support equipment (such as air conditioning, lighting, or other).

Most data centers today consider a target value of 1.5 good, with some companies such as Google trying to drive their PUE below 1.2 – an industry benchmark.

Other data centers are not even at the point where they can collect meaningful PUE data. The previous Google link has an extended description of data collection methodology, which is a great introduction to the concept. The Uptime Institute of course has a large amount of support materials. And a handy Bong search reveals another 995,000 results on the topic. No reason why any data center operator should be in the dark or uniformed on the topic.

So let’s use a simple PUE example and carbon calculation to determine the effect of a poor PUE:

Let’s start with a 4 MW data center. The data center currently has a PUE of 3.0, meaning of the 4 MW of power consumed within the data center 3MW are consumed by support materials, and 1MW by actual IT equipment. In California, using the carbon calculator, this would return 357 tons of carbon produced by the IT equipment and 1071 tons of carbon produced by support equipment such as air conditioning, lights, poorly maintained electrical equipment, etc., etc., etc…

1071 tons of carbon each month, possibly generated by waste which could be controlled through better design, management, and operations in our data centers. Most commercial data centers are in the 4~10MW range. Scary.

The US Department of Energy recently did an audit entitled “Department of Energy Efforts to Manage Information technology in an Energy-Efficient and Environmentally Responsible Manner,” which highlights the fact even tightly regulated agencies within the US Government have ample room for improvement.

“We concluded that Headquarters programs offices (which are part of the Department of Energy’s Common Operating Environment) as well as field sites had not developed and/or implemented policies and procedures necessary to ensure that information technology equipment and supporting infrastructure was operated in an energy-efficient manner and in a way that minimized impact on the environment.” (OAS-RA-09-03)

What Can We Do?

The easiest thing to do is quickly replace all traditional lighting with low power draw LED lamps, and only use the lamps when human beings are actually within the data center space working. Lights generate a tremendous amount of heat, and consume a tremendous amount of electricity. Heat=air-conditioning load if that wasn’t already obvious. Completely wasted power, and completely unnecessary production of carbon. If you are in a 10,000sqft data center, you may have 100 lighting fixtures in the room. Turn them off.

If your data center requires security cameras 24×7, consider using dual-mode cameras that have low light vision capability.

Place blanking panels in all cabinets. Considering removing all open racks from your data center unless you are using them for passive cabling, cross-connects, or very low power equipment. Consider using hot or cold aisle containment models for your cabinet lineups. Lots of debate on the merits of hot aisle containment vs. cold aisle containment, but the bottom line is that cool air going into a server makes the server run better, reduces the electrical draw on fans, and increases the value of every watt applied to your data center.

Consider this – if you have 10 servers using a total of 1920 watts (120v with a 20 amp breaker <at 16 amps draw>), that gives you the potential of running those 10 servers at full specification draw. That includes internal fans which start as needed to keep internal components cool enough to operate within equipment thresholds. If the server is running hot, then you are using your full 192 watts per server. If the server is running with cool air on the intake side, no hot air recirculation producing heat on the circuit boards, then you can reasonably expect to reduce the electrical draw on that component.

If you are able to reduce the actual draw each server consumes by 30~40% by removing hot air recirculation and keeping the supply side cool, then you may be able to add additional servers to the cabinet and increase your potential processing capacity for each breaker and cabinet by another 30~40%. This will definitely increase your efficiency, cost you less in electricity and power, give you additional processing potential.

Sources of Information

Quite a few sources of information, beyond the Bing search are available to help IT managers and data center managers. APC probably has the most comprehensive library of white papers supporting the data center discussion (although like all commercial vendors, you will see a few references to their own hardware and solutions). HP also has several great, and easy to understand white papers, including one of the best reviewed entitled “Optimizing facility operation in high density data center environments” – a step-by-step guide in deploying an efficient data center.

The Bing search will give you more data than you will ever be able to absorb, however the good news is that it is a great way to read through individual experiences, including both success stories and horror stories. Learn through other’s experiences, and start on the road to both reducing your carbon footprint, as well as getting the most out of your data center or data center installation.

Give us your opinions and experiences designing and implementing the green data center – leave a comment and let others learn from you too!

John Savageau, Long Beach

Tagged with: carbon • carbon footprint • data center • green • pue

Telecom Risk and Security Part 4 – Facilities

On October 14, 2009, In Internet and Telecom, By Administrator

A 40 year old building with much of the original mechanical and electrical infrastructure. A 40 year old 4000 amp, 480 volt aluminum electrical buss duct, which had been modified and “tapped” often during its life, with much of the work done violating equipment specifications. With the old materials such as buss insulation gradually deteriorating, the duct expanding and contracting over the years, the fact aluminum was used during the initial installation to either save money or test a new technology vision – it all becomes a risk. A risk of buss failure, or at worst a buss failing to the point it results in a massive electrical explosion.

Facility Explosion Sound extreme? Now add a couple of additional factors. The building is a mixed use-telecom carrier hotel, with additional space used for commercial collocation and standard commercial office space. This narrows it down to most of the carrier hotel facilities in the US and Europe. Old buildings, converted to mixed-use carrier hotel and collocation facilities, due mainly to an abundance of vacant space during the mid-1990s, and a need for telecom interconnection space following the Telecommunications Act of 1996.

Over the past four years the telecom, Internet, and data center industry has suffered several major electrical events. Some have resulted in complete facility outages, others have been saved by backup systems which operated as designed, preventing significant disruption to tenants and the services operated within the building.

A partial list of recent carrier hotel and data center facility outages or significant events include some of the most important facilities in the telecom and Internet-connected industry:

365 Main in San Francisco
RackSpace hosting facilities in Dallas
Equinix facilities in Australia and France
MPT in San Jose
IBM facility in NZ
Fisher Plaza in Seattle
Cincinnati Bell

And the list goes on. Facilities which are managed by good companies, but have many issues in common. Most of those issues are human issues. The resulting outages caused havoc or chaos throughout a wide range of commercial companies, telecom companies, Internet services and content.

The Human Factor in Facility Failures

Building a modern data center or carrier interconnection point follows a fairly simple series of tasks. Following a data center design and construction checklist, with strict compliance to the process and individual steps, can often mean the difference between a well-run facility and one that is at risk of failure during a commercial power outage, or systems failure.

In the design/construction phase, data center operators follow a system of:

Determining the scope of the project
Developing a data center design specification based on both company/industry standards
Designing a specific facility based on business scope and budget, which will comply with the standard design specification
Publish the design specification and distribute to several candidate construction management companies and engineering companies
Use a strong project manager to drive the construction, permitting, certification, and vendor management process
Complete systems integration and commissioning prior to actual operations

Of all the above tasks, a complete commissioning plan and integration test is essential to building confidence the data center or telecom facility will operate as planned. Many outages in the past have resulted from systems that were not fully tested or integrated prior to operations.

Facility Checklist An example may be a breaker coordination study. This is the process of ensuring switch gear and panel breakers from the point of electrical presentation by the local power utility down to individual breaker panels are set, tested, and integrated according to vendor specification. Without a complete coordination study, there is no assurance components within an electrical system will either operate correctly during normal conditions, or operate correctly during equipment failures. An essential component of a complete systems integration test. Failure to complete a simple breaker coordination study during commissioning has resulted in major electrical failures in data centers as recently as 2008.

The InterNational Electrical Testing
Association (NETA) provides guidance on electrical commissioning for data centers under “full design load” conditions. This includes testing recommendations to test performance and operations including the sequence of operations for electrical, mechanical, building management systems/BMS, and power monitoring/management. The actual levels of NETA testing are:

Level 1- Submittal Review and Factory Testing
Level 2- Site Inspection and Verification to Submittal
Level 3- Installation Inspections and Verifications to Design Drawings
Level 4- Component Testing to Design Loads
Level 5- System Integration Tests at Full Design Loads

No company should consider collocation within a facility that cannot produce complete documentation that integration testing and commissioning was completed prior to facility operations – and that testing should be at NETA Level 5. In some cases, documentation of “retro” testing is acceptable, however potential tenants in a facility should be aware that is still a compromise, as it is almost impossible to complete a retro-commissioning test in a live facility.

Bottom Line – even a multi-million dollar facility has no integrity without a detailed design specification and complete integration/commissioning test.

The Human Factor in Continuing Facility Operations

Assuming the facility adequately completes integration and commissioning at NETA Level 5, the next step is ensuring the facility has a comprehensive continuing operations plan to manage their electrical (and mechanical/air conditioning) systems. There are two main recommendations for ensuring the annual, monthly, and even daily equipment maintenance and inspection plans are being completed.

Computerized Maintenance Management System (CMMS)

Data centers and central offices are complex operations. Thousands of moving parts, thousands of things that can potentially break or go wrong. A CMMS system tries to bring all those components together into an integrated resource that includes (according to Wikipedia)

Work orders: Scheduling jobs, assigning personnel, reserving materials, recording costs, and tracking relevant information such as the cause of the problem (if any), downtime involved (if any), and recommendations for future action
Preventive maintenance (PM): Keeping track of PM inspections and jobs, including step-by-step instructions or check-lists, lists of materials required, and other pertinent details. Typically, the CMMS schedules PM jobs automatically based on schedules and/or meter readings. Different software packages use different techniques for reporting when a job should be performed.
Asset management: Recording data about equipment and property including specifications, warranty information, service contracts, spare parts, purchase date, expected lifetime, and anything else that might be of help to management or maintenance workers. The CMMS may also generate Asset Management metrics such as the Facility Condition Index, or FCI.
Inventory control: Management of spare parts, tools, and other materials including the reservation of materials for particular jobs, recording where materials are stored, determining when more materials should be purchased, tracking shipment receipts, and taking inventory.
Safety: Management of permits and other documentation required for the processing of safety requirements. These safety requirements can include lockout-tagout, confined space, foreign material exclusion (FME), electrical safety, and others.

And we can also add additional steps such as daily equipment inspections, facility walkthroughs, and staff training.

SAS 70 Audits

The SAS 70 Audit is becoming more popular with companies to force the data center operator to provide audited documentation by a neutral evaluator that they are actually completing the maintenance, security, staffing, and permitting activities as stated in marketing and other sales negotiations.

Wikipedia defines a SAS70 Audit as:

“… the professional standards used by a service auditor to assess the internal controls of a service organization and issue a service auditor’s report. Service organizations are typically entities that provide outsourcing services that impact the control environment of their customers. Examples of service organizations are insurance and medical claims processors, trust companies, hosted data centers, application service providers (ASPs), managed security providers, credit processing organizations and clearinghouses.

There are two types of service auditor reports. A Type I service auditor’s report includes the service auditor’s opinion on the fairness of the presentation of the service organization’s description of controls that had been placed in operation and the suitability of the design of the controls to achieve the specified control objectives. A Type II service auditor’s report includes the information contained in a Type I service auditor’s report and also includes the service auditor’s opinion on whether the specific controls were operating effectively during the period under review.”

Many companies considering outsourcing within the financial services industries are now considering a SAS70 audit essential to considering candidate data center facilities to host their data and applications. Startup companies with savvy investors are demanding SAS70 audits. In fact, any company considering outsourcing their data or applications into a commercial data center should demand to obtain or review SAS70 audits for each facility considered.

Otherwise, you are forced to “believe” the words of a marketer’s spin, a salesman’s desperate pitch, or the words of others to provide confidence your business will be protected in another company’s facility.

You Have the Best Data Center One thing to keep in mind about SAS70 audits… The audit only reviews items the data center operator chooses to audit. Thus, a company may have a very nice and polished SAS70 audit documentation, however the contents may not include every item you need to ensure the data center operator has a comprehensive operations plan. You may consider finding an experienced consultant to review the SAS70 document, and provide any additional guidance on whether or not the audit actually includes all facility maintenance and management items needed to ensure continuing protection from mechanical, monitoring/management, electrical, security, or human staffing failures.

Finally, Know Your Facility

Facility operators are traditionally reluctant to show a potential customer or tenant their electrical and mechanical diagrams and “as-built” documentation for the facility. This is the point you would find a 40 year old aluminum buss duct, single points of failure, and other infrastructure designs and realities you should know before putting your business into a data center or carrier hotel.

So, when all other data center and carrier hotel facilities appear equal, in geography and interconnections, look at facilities which will incur the least impact if your interconnections are disrupted, and demand your candidate data center operator and hosting provider are able to provide you complete documentation on the facility, commissioning, CMMS, and SAS70.

Your business, the global marketplace, and network-connected world depend on forcing the highest possible standards of facility design and operation.

John Savageau, Long Beach

Telecom Risk and Security Part 3 – Human Factors

On October 13, 2009, In Government Regulatory, Internet and Telecom, By Administrator

An employee enters the meet-me-room at a major carrier hotel in Los Angeles, New York, or Miami. He is a young guy recently graduated from high school, hired to do cable removal for circuit disconnects at minimum wage. Although young, he has a wife and child, and has recently been fighting with in-laws over his ability to support a family. Frustration and anger overcome his emotions, and he turns to the ladder rack jammed with cable and starts hammering at the cables for all he is worth.

Network operations centers around the world see circuits dropping, and customers with critical financial, military, Internet, and broadcast news services are shut down. In the space of about one minute our young employee has taken down several thousand individual circuits, creating near chaos in the global telecommunications community.

In their report on Trusted Access to Communications Infrastructure, the NSTAC Vulnerabilities Task Force advises “”it is important to recognize that any one individual with malicious intent accessing any critical telecommunications facility could represent a threat. The threat of insiders performing malicious acts also transcends each type of site discussed in this document.”

Security in Telecom The event noted in part 2 of this series describing the outages in Northern California following damage to a manhole housing telecom was real. The resulting disruption to regional communications was a wakeup call to the telecom community, law enforcement, and communities affected. It is clear the perpetrator knew what he was doing, and knew exactly what vulnerabilities the major telecom companies had which he could exploit.

There have been many other cases such as Level 3 Communications loss of a major core router in 2006 supporting regional Internet services in London due to theft, a break-in at BT’s switching facility in Birmingham during the same period resulting in the loss of thousands of telephone lines, showing this is not just an American problem, but a global vulnerability.

The message is clear, as an industry our most obvious threat to information and communications security is not a natural disaster, it is people with industry knowledge or access to our critical facilities.

The Telecom and Data Center/Carrier Hotel Industry’s Role in Managing Human Security Risks

Data centers and central offices are in a constant state of change, maintenance, and growth. While facility network operations staff are generally long term employees, with a history of employment and performance, many others entering our data centers are not well known to the landlord.

Janitorial and maintenance staff are normally contracted to vendors, mechanical and electrical workers are contracted to maintenance and engineering companies, and construction contractors often use temporary staff from agencies such as “Labor Ready” and other day labor companies. In most cases data center or landlord employees are given a cursory background check prior to employment, however others entering even critical areas within the data center or central office meet-me-room may be entirely unknown to the facility.

While normally under some level of supervision, or access management, contractors, maintenance people, and even data center tenants are often free to move around the facility without direct security observation. As shown above, it would only take an angry, disgruntled, or undisciplined person seconds to cause a major calamity in our global communications system.

In a worst case, that person may be a terrorist with a detailed plan to cause damage to the facility once given even minimal access. High voltage electricity, water systems from cooling infrastructure, or access to switching equipment and cable interconnections are all exposed within the data center, and any element could be used to cause a major disruption within the meet-me-room or data center.

Most carrier hotels are located in “mixed-use” buildings, in high-rises with additional tenants who may not even be in the data center or telecom industry. This compounds the problem, as those tenants are often reluctant to comply with security and access requirements at the level of a critical telecom facility.

The issue becomes even more acute when we realize that much of the infrastructure supporting carrier hotels transits “risers” between floors, often through floors occupied by non-telecom tenants who may have physical access to riser space within their offices.

Secure Your Manhole Covers There are a few data centers within the United States where security is comprehensive enough to reduce the risk of malicious intent to a very low level. While many tenants find the access and supervision within the facility extreme, facility resources are protected from all but the most aggressive vandalism or attack.

The NSTAC recommends that in the US the telecom industry establish best-practices guidelines to screen personnel prior to unescorted or unrestricted access to critical facilities, such as carrier hotels and carrier central offices. This may include a national agency check to ensure the person requesting access does not already have a profile indicating they could potentially be a threat to the facility.

The US government may give this additional support, as much of the US government, state, and local communications services are supported either in carrier central offices or commercial carrier hotels.

Recommendations for the Communications Industry

While it is clear not all persons entering a data center or carrier hotel facility can be completely screened, there are tasks each carrier and commercial data center operations should complete. Those could include:

Complete background checks for all direct employees
Pre-employment screening which would include a personality profile (indicating if they are in a high risk category for emotional stress)
Supervision of all contractors on site by a direct company employee who is aware of the risk posed for each type of equipment in proximity to the contractor (such as electrical equipment <UPS, breaker panels, switchgear, chilled water pipes, etc>)
Training in situational awareness – being able to identify activities not normal for others in your facility
Cooperation with law enforcement and other agencies
Working with industry groups to create and follow an industry “best practices” for facility security and human resource management
Ensure at least in the streets and areas immediately adjacent to the facility all manhole covers and utility entry points are locked and secured, preventing persons from accessing telecom, electrical, and water infrastructure supporting the building

“Unfortunately our most likely enemies will throw explosives into unguarded cable interconnect rooms or drop cans of petrol into unlocked manholes. End of Cyber War. You might characterize this as the provenance of a 23 year old fundamentalist Skywalker with a cell phone modem and a wild-eyed cousin in Munich figuring out how to blow up the Internet Death Star and stop Predator attacks on his village. Totally asymmetric dude! (From Bob Fonow’s “The Death Star?: Cyber Security vs. Internet security”)”

The commercial operators of data centers and carrier hotels have a tremendous responsibility not only to their owners and shareholders, but also the global telecom community and global economic community. The potential impact, even in the short term to a malicious attack on a meet-me-room at One Wilshire, 60 Hudson, the Westin Building, Telehouse in London, or the NAP of the Americas would be immediate, and extremely disruptive.

Human factors are the threat. Let’s not forget the lessons learned over the past couple years, and keep diligent, have good human situational awareness, and understand the sense of urgency we must apply to ensuring our communications infrastructure is secure.

Let us know your opinions, experiences, and recommendations

John Savageau, Long Beach

Previous articles in this series

Tagged with: 60 hudson • carrier hotel • human factor • one wilshire • security • telecom • telehouse • westin

Telecom Risk and Security Part 2 – The Carrier Hotel SuperNode

On October 12, 2009, In Government Regulatory, International ICT, Internet and Telecom, LA Life, cloud computing, By Administrator

February 1996. A half-ton bomb planted in a small truck near South Quay Station close to the recently renovated commercial district of Canary Wharf. The bomb detonated around 1900 hours, bringing down a six story building, and severely shaking Canary Wharf Tower and other buildings around the Docklands area. The area, home to much of the telecommunications interconnection capacity connecting the UK and Europe to the rest of the world, is severely damaged and all surrounding activity disrupted.

Today the Docklands area continues to support many important, high density communications interconnection points, including Telehouse Europe, the London Internet Exchange (LINX), and the London Network Access Point (LONAP) – in addition to individual nodes and facilities operated by European and other international telecommunications carriers.

This includes companies operating submarine fiber optic cable systems. These densely interconnected areas are referred to as telecommunications “SuperNodes,” or if the facilities are located at individual facilities, “Carrier hotels.”

A Global Issue The US National Security Telecommunications Advisory Committee (NSTAC) defines a carrier hotel (or SuperNode) as “conditioned floor space operated by a commercial landlord for the purpose of hosting multiple service providers.” The most well-known supernodes are 60 Hudson in New York City, The NAP of the Americas in Miami, One Wilshire in Los Angeles, and the Westin Building in Seattle.

Carrier hotels emerged in the late 1900s following the Telecommunications Act of 1996, which required US incumbent carriers to provide interconnection or collocation space for the new competitive carrier industry. The problem for the carriers, and opportunity for commercial building owners, was one of the carrier facilities exhausting available space.

The commercial landlords were able to provide building space, partially due to low occupancy in city center areas near large carrier central offices (such as Bunker Hill in Los Angeles) during the late 1990s, and competitive carriers were able to build out their interconnection infrastructure with little or no interference by the incumbent carriers.

Carrier hotels can also be considered “scale free,” with the only real limitation on growth being the physical space available within a property, as well as electricity and cooling for electronics and switching equipment. This may not even be a large problem, as much of the carrier hotel interconnection volume is done through “passive” cross connects. Cross connects fiber optic to fiber optic splicing which does not require local electronics, and thus is not directly vulnerable to cooling and power issues.

What is the Impact of Losing a Carrier Hotel or SuperNode?

Could another attack similar to the 1996 Docklands incident potentially have the impact of severing interconnection capacity between communications carriers, Internet service providers, and news or information resources?

The extent of disruption would depend on the amount of switching and multiplexing equipment and physical interconnection capacity each company locates within the Telehouse facility, or the immediate area.

This is a source of much debate. In the US, nearly all facility-based (own their own cable) carriers and large virtual carriers have numerous interconnection sites located throughout the country. The loss of a single node or interconnection facility would not significantly disrupt national or international communications.

The Federal Communications Commission provides guidelines for facility-based carriers through the Network Reliability and Interoperability Council (NRIC) which advises “carriers place a high priority on service reliability by building networks with alternative routes, backup facilities, and other assurance capabilities.”

The danger at the SuperNode or carrier hotel is not necessarily one of the incumbent or long distance facility-based carrier. It is more an issue with:

International carriers with only one or two physical landing points in North America (or Europe)
Local exchange carriers with limited interconnection capacity outside of the carrier hotel
Internet service providers operating in a smaller geography (Tier 3 access networks)
Hosting companies and content delivery providers with single or limited Internet access
Local fiber providers with limited diversity within a city center

This is actually quite alarming. When you start to consider the outsourcing industry, including cloud computing, entertainment, and the number of companies who do not have strong disaster recovery plans – including geographic diversity within their applications and communications access - the potential for disruption is high.

Most of the SuperNodes provide interconnections for more than 200 facility-based carriers, networks, content providers, cloud service providers, and other hosting or business outsourcing. Understanding the reality that we live in a very global economy, losing interconnection capacity of even one SuperNode could render a large percentages of the global financial, logistics, business-to-business, disaster response, and government communications inoperable for hours of days while restoral plans are either implemented or conceived.

Companies with hosted applications and data center presence either in or near the failure point could be isolated or destroyed. Hosted companies “single-threaded” with one carrier connection that using the carrier hotel for its main interconnection point would be shut down.

The bottom line, companies without a strong restoral, backup, disaster recovery, and physically diverse network will suffer a catastrophic failure of their systems, with the length of outage entirely dependent on the facilities ability to recover from an outage or failure.

If more than one SuperNode is disrupted, such as all facilities on the US West coast, international communications both on Internet links (the majority of international communications today) and dedicated capacity will cause significant damage and disruption to both US and international communications.

What Can Cause a Major Failure?

There are many factors to consider, both human and natural, when looking at global communications infrastructure. Just in the past 5 years we’ve seen significant submarine cable disruptions due to both undersea earthquakes and cable cuts to strong waves hitting cable landing facilities on the coasts. Carrier hotels are primarily located on the coasts, in large cities, due to the proximity of both submarine cables supporting international communications, and the fact most North American and European terrestrial cable routes tend to interconnect at major coastal cities.

Coastal cities are vulnerable to:

Earthquakes
Typhoon and Hurricane wind/storm swells
Tsunami
Tropical rain and flooding

Human factors are also a concern, with potential problems such as:

Civil disorder
Terrorist attack
Vandalism
Employees (disgruntled, human error, etc)

If you look at the streets adjacent to buildings such as One Wilshire, you can see the evidence of dozens of carrier tags trying to mark and protect their conduit routes running through the streets, and entering the carrier hotel facility at One Wilshire. Few of the manholes around the area are locked, and few if any local building security officers or police officers will ever challenge a company setting up a couple of traffic cones and entering the manhole.

The potential for human disruption, just by having access below the street level near a building such as One Wilshire or 60 Hudson could be extreme. From below ground potential terrorists have access to power substations, water lines, and hundreds of conduits supporting the entire metro area – including the carrier hotel. A well placed explosive below grade in downtown Los Angeles could potentially disrupt the communications of more than 450 network and Internet-connected companies operating within One Wilshire or immediately adjacent buildings.

Many of the carrier hotels do not have battery backup or even redundant power, as the “meet-me-rooms” fell under the “scale free” rapid growth in the late 1990s and 2000s when those rooms had little or no management, admistrative controls/regulation. This is gradually being brought under control in the largest facilities, and most smaller facilities such as the NAP of the Americas in Miami are very well controlled.

This was proven possible during the 1996 attack in London, and could occur again at any single, or multiple carrier hotel facilities located in the United States and other countries. It is a real problem, and one that is not lost on governments around the world.

What We Can, and Are Doing to Protect Our Communications Assets

The key to all applications and communications security is diversity and redundancy. Very few submarine cables are being built today without at least a diverse loop, or a restoral agreement with a competitive cable company. If there is a single location or cable disrupted across the oceans, and restoral capacity is planned, the problem can be managed.

For North American carriers and Internet Service Providers, having a network with multiple “peering” points in different geographic locations will minimize disruption, and in the case of most regional and global networks that is the case. In fact, most large Internet networks require interconnections in multiple locations before they will consider “peering” relationships. That is of course for both traffic management, as well as disaster planning.

This would mean an Internet Service Provider would best plan their network for both physical high capacity interconnections in multiple carrier hotels, but also peering or disaster peering plans for interconnecting at public peering points, such as PAIX, Any2, Equinix, and Telehouse in the US, or other major Internet Exchange Points (IXPs) in London, Amsterdam, Frankfurt, and other Asian cities.

For those carriers and ISPs planning long distance interconnections, care must be taken to ensure route diversity. In some cases, multiple carriers will purchase capacity on a wholesaler fiber provider’s infrastructure (such as Level 3 Communications, XO, and Time Warner), with the possibility several different network providers will buy capacity on their long distance route using the same cable system.

In many cases, such as cable landing stations dotting Long Island in New York, the actual cable connecting those facilities to the carrier hotels and their own cable capacity management facilities follow a single route. The risk is that a single backhoe, terrorist, or vandal could potentially cause serious international communications damage by simply cutting a trough across the roadway, or jumping into a manhole and cutting cable.

“Vandals are to blame for the massive phone and Internet outage in Silicon Valley on Thursday, an AT&T representative has confirmed.” (CNET News, 9 Apr 2009)

An incident in early 2009 near San Jose (California) where an individual performed a similar act of vandalism caused significant disruption across a large area in Northern California. The above story confirms the danger present when critical infrastructure is not adequately protected, and a single person can enter a manhole with the potential of such widespread impact.

Physical cable and route diversity guarantees should be part of every disaster recovery and route planning negotiation.

Those companies outsourcing their mission and company-critical data and applications must look at geographic diversity, with the ability to dynamically restart applications with industry and customer-acceptable recovery point and recovery time objectives. Cloud computing technology is getting closer to providing this for the future, but not quite ready for offering service level objectives.

The US Government Weighs In

The NSTAC believes the government should work with private industry to develop both operational best practices, as well as a solid, coordinated, threat warning system to assist carrier hotel, data center, and SuperNode operators to ensure the best level of security for national and global infrastructure.

Police departments should have some level of visibility into carrier hotels and SuperNodes, data centers, and telecommunications company central offices. Not because we want “big-brother” looking into our business, but because we want law enforcement to understand the nature of our telecom business, and what could potentially happen if human beings are able to damage local infrastructure (which includes emergency responder infrastructure).

The NSTAC recommends individuals employed at carrier hotels and critical infrastructure facilities go through an initial security check. This may be in part because the national authorities probably have either own communications running through SuperNodes, and have recognized there is a reasonable chance US government and military communications could also be damaged or disrupted in the event of a facility failure or loss.

The FCC and NSTAC also recognize the burden of responsibility ultimately falls on the individual networks and customers. Our economy and communications infrastructure depend on each company having good disaster recovery and diversity plans. Individual users must ensure we get service level agreements with a clause ensuring physical route diversity in backup and DR site interconnections.

ISPs need to multi-home their networks. Not just at a single interconnection point, carrier hotel, or IXP – but in separate facilities, preferably in separate geographies.

The government is working with representatives from the telecom, vendor (electronic switching equipment, etc), applications, business community, and government agencies on a continuing basis to ensure US policy is kept current, and the threat/risk of our current infrastructure is understood. The President’s National Security Telecommunications Advisory Committee (NSTAC) is now part of the US Department of Homeland security, and coordinates much of the discussion.

As users, we need to take action as well. We can do any or all of the following to ensure not only our security in global communications, but also at our businesses and home:

Ask your hosting provider if they have a disaster recovery plan – Get proof
Ask your network provider if they are multi-homed and multi-homed in multiple geographies – Get proof
Ask your provider if their physical diversity is using physically separate fiber routes
Ask your hosting provider if they have good coordination with law enforcement for local security – Get proof
Ask your international VPN (virtual private network) provider if their cable system has a restoral plan, or if you have geographic fail-over on a separate cable – Get proof

In short, the burden is ultimately on the end user to ensure their business or activity survives a major disaster. We must drive our vendors, and should seriously consider strongly supporting greater regulation and oversight of our critical infrastructure facilities to ensure we do not lose a resource that could potentially contribute to a global economic and communications catastrophe.

What are your concerns? Do you believe we are OK in our current telecom environment? Should we do more? Your comments are welcome.

John Savageau. Long Beach

Pacific-Tier Communications

Technology Innovation Topics

Shrouding the Net Neutrality Debate in a Cloud of Politics

Managing Emergency Communications – A Tutorial with Marc Ladin

Deleting Your Hard Drives – Entering a Green Data Center Future of SSDs

Recent and Fun California Green Energy Initiatives

How Green is Your Data Center?

Telecom Risk and Security Part 4 – Facilities

Telecom Risk and Security Part 3 – Human Factors

Telecom Risk and Security Part 2 – The Carrier Hotel SuperNode

Categories

Recent Posts

Pages

Article Archive