Pacific-Tier Innovators

I first met Mark Fishburn at the Convergence Technology Council (CTC) in Calabasas, California. Mark was a director in the organization, and had very strong ideas about networking and Ethernet. Going beyond the standard role we all play at professional networking venues, he distinguished himself from the group by presenting a passion for teaching others, and presenting his ideas in language nearly anybody could easily understand. Mark was always easy to find at CTC meetings, as he was the center of the largest groups of people who wanted to hear what he had to say.

Mark is a true innovator, and generates a lot of inspiration among CTC members with his visions and thought leadership in a variety of technology and business-related topics. I met Mark in Tarzana, California, to learn more about his vision related to Carrier Ethernet, as well as to gather some advice for entrepreneurs.

Pacific-Tier: Mark, tell us a little about yourself. How did you come to the San Fernando Valley, and what do you do?

Mark Fishburn: I worked at US companies for many years when I was in London, and one day I said I could fix a (problem) in the US headquarters, and they said “OK.” So I came across as a corporate officer in a company called Retix. I worked with them for a while, and then started my own company.

So that was my business, a software company, and then back into data communications, and worked for a company called NetCom Systems, which then became a company called Spirent.

Pacific-Tier: You’ve been involved with the Metro Ethernet Forum for quite some time. What interested you about the MEF?

Mark Fishburn: Well it actually goes back some time to my interest in Ethernet, and the world of Ethernet from the very early days. in 1982 I installed my first Ethernet system while working for Xerox, and that was in Paris. it was one of the very first Ethernet installations.

And as a result of that I gathered a great interest in Ethernet. In the old times, working for an Ethernet test-equipment company, we put out on e of the first fiber Ethernet products, and a few years later one of the first copper Gigabit Ethernet products.

And so it went on. I was intimately involved as chairman with the 10 Gigabit Ethernet Alliance, and the Gigabit Ethernet Alliance before that. It became apparent this was all triggered by the definition of fiber Ethernet. It really reached out beyond the boundaries of local area networks to the metro network.

That really paved the way for Ethernet services to be provided by service providers, and not just live inside the LAN. That was really the initial foundation of the Metro Ethernet Forum/MEF.

It was all about, really advancing the adoption of optical-based fiber Ethernet.

Pacific-Tier: I guess that brings us up to your current venture, which is CENX. Can you tell us anything about CENX?

Mark Fishburn: Sure, let me just give you a bit of background, because it is all really very connected.

In the substantiation of the MEF it became clear there were many different technologies that were or could be connected together using the Ethernet as a ghost in the machine.

And thus were born Ethernet services. And in 2004 carrier Ethernet was created and defined by the MEF by providing ubiquitous services worldwide independent of the service providers providing them, and also the equipment it is connected on.

And that really led to development of the need to have global connections between the service providers who are providing these Ethernet carrier services.

Although I say that in a sentence, it actually took about eight years to transpire and it led to a business that in 2009 has become about a $20 million global services revenue.

At this point in time, as these networks have grown, there is a requirement to connect more of them together in a way which preserves the differentiation of the service providers and creates a global (Ethernet) interconnectivity.

That really led to the formation of our company CENX (Carrier Ethernet Neutral Exchange) which was established to created, effectively a service-level interconnect between the service providers worldwide, and negate the enormous cost and pain in making those connections possible.

Pacific-Tier: Excellent. It’s kind of a sketchy economic environment, a tough time for businesses. What drove you to start a new business in this tough economy?

Mark Fishburn: Well, there are some areas that grow in spite of the economic downturn. The areas that grow are those that potentially save cost, or those that are pushing the envelope and generating more revenue.

Carrier Ethernet is such an animal. It (the industry) grew somewhere around 33% last year in America alone. So while the economy is growing people look for significantly more economic ways to effectively use the same old applications, while paving the way for new applications data driving mobile technology.

So, in this economy to do that was both a natural, and almost necessary step to advance this industry. And as such it was pretty natural for those people who realize this to be attracted to our company, to invest in it, and to meet that need.

Just like anything else, if you have a sufficiently difficult problem, and there is a need to solve it, it save money, and helps make money for people, and makes their job easier, then it’s a very compelling case.

Pacific-Tier: You’ve been a director with the Convergence Technology Council of California/CTC here in the San Fernando Valley (Los Angeles) providing thought leadership and help to a lot of people who are members. What advice do you have for people who may be having trouble with their jobs, been laid off, or are young graduates getting ready to enter the workforce – is there hope for entrepreneurs and those getting ready to jump into the technology industry?

Mark Fishburn: I would say absolutely. I think this is a great time to start a new venture. If you look at every great new venture, this has been repeated many times. In all the great companies that were founded – they weren’t founded when the economy was good, they were founded when there were significant problems that gave people an opportunity to really look at the idea that necessity is the mother of invention.

It’s like anything else, there are tremendous opportunities, still driven by technology, or different social climates driving the way people communicate now, rather than the way they did before. So within technology is really an unlimited opportunity for people to look at an issue, or to realize their dream and go for it.

Pacific-Tier: Young people today, they have technology diffused into their education, and into their childhood and youth at a rate that we never had in our middle-aged years. How do you feel about the youth today? Are they going to be able to take this thing that we’ve built and make it better?

Mark Fishburn: I kind of look at it a little differently. I think in a way they are driving it. Because if you look at somebody who is multi-tasking, if you look at the corporate world of maybe a couple years ago, well when you were at work you were at work. When you went home you played.

It’s become so blurred that the distinction between work, collecting information, entertainment, and communications, it is going to happen in a way that is connected 24 hours, and I think that young people today are living in t
he world of communications – in a way that they communicate with each other, in a way they focus, in a way that they are constantly multi-tasking and moving towards whatever is the next and most convenient way to gather.

So I believe that the youth of today is programmed into this multi-processing environment that they have, and that it’s way (young) people operate, doing multiple things at the same time, is the way of the future, and I believe that people who have been brought up in the world with mobile technology and communications, texting and talking, thinking and playing – all of those at the same time. I think all of those things are the wave of the future.

I think entrepreneurs who connect to that will do well.

Pacific-Tier: That’s very encouring. Thank you today for your counsel, great advice, stories, and great talk!

Mark Fishburn: Sure – can I add one more thing?

Pacific-Tier: Of course!

Mark Fishburn: I would say that one of the things that really led me to doing this was the realization that a lot of people would fear to go into something new like this, or to start a new job. But the alternative is unpalatable. Surviving until you die is no way forward. And I believe that if you are passionate about something that you really have nothing to lose by trying it out.

If you don’t do that, you might regret it forever. So I would say, just go for it.

You can contact Mark at mark@cenx.com

Check out the entire Pacific-Tier Communications Innovators and Entrepreneur Series

I first met Matt Hiles while he was director of business development with Looking Glass Networks in Los Angeles. As a customer looking for telecom services, navigating the providers, technologies, and deal structures can be confusing. Matt took the time to explain all aspects of the business, cost structures, and how he would get us a great deal – while still making money for his company. Matt stood out alone from a world of "wheeling and dealing" telecom sales people, unique in providing the customer a level of confidence they were getting the best product, for the best price, with the best service.

Pacific-Tier: Today we have Matt Hiles, managing partner with Mosaic Networx. Hello Matt! So tell us a little about yourself, how did you get into this business?

Matt Hiles: I started in telecommunications right out of college, and I’ve been in the business, in one form or another, since – which is about 20 years. I’ve been in a variety of telecommunications, voice, and service providers. I’ve also spent a period of time in the data center side of the industry as well.

Pacific-Tier: now you are with Mosaic Networx. Can you give a little background on Mosaic. What are you, what do you do, and what type of business problems do you solve?

Matt Hiles: Mosaic Networx is a carrier neutral, data services provider. We provide a supply chain management service primarily for enterprise companies, but secondarily to wholesale providers and telecommunications providers. From a supply chain management perspective we provide a value add in three functional areas which are pricing, procurement, and provisioning.

What we’ve found is that, in the enterprise space, there is a lack, or need in one of those areas. Typically all of those areas. Where enterprise decision-makers and IT managers don’t have the depth and breadth of knowledge of the telecommunications providers and options that are available.

So we price them, then procure them, provision them, and then manage them ongoing on the back end.

Pacific-Tier: Well, that’s pretty cool. So who is your market, who would be your customer?

Matt Hiles: Our customers are small, medium, and we even have several Fortune 500 companies. We have a strong vertical in the financial services market. Specifically we work with the low-latency, high frequency trading guys. We’ve also worked with public wholesale companies who may not have the buying power we have, so we add some pricing value for those types.

Pacific-Tier: I’ve noticed you are based in Long Beach, California. Other people in your company are scattered around the United States, with diverse locations for your primary management team – does that provide you any challenges?

Matt Hiles: I imagine it provides some challenge, although It would be hard to quantify them. We haven’t really seen them. I think where we’ve done an outstanding job in is finding the right people.

We have 18 personnel in the functional areas in the company, whether its finance or operations, or on the sales side as well. So the distributed environment that we have seems to work out just fine.

Would we have a little bit more camaraderie in a common office? Probably.

Pacific-Tier: So it’s rather tough economic time right now. We’ve had kind of a sketchy run over the last year. What motivated you to start up a company in the last year or so and how do you feel about being an entrepreneur in a tough economic environment?

Matt Hiles: So, I suppose that timings everything, right? We didn’t know we would start a company in a tough economic period. But, the economy notwithstanding , I think there is always business. And for innovative entrepreneurs who can go out and create value for customers, provide them an outstanding customer experience, then good or bad times I think you can be successful.

Pacific-Tier: So what advice do you have for other entrepreneurs, graduates who are looking at a tough economy, what advice do you have for other budding visionaries and entrepreneurs?

Matt Hiles: I think you have to have an expertise. It doesn’t make a lot of sense in my mind to venture into an area as an entrepreneur where you don’t have years of background and can consider yourself a subject-matter expert. I think that is (not being a subject-matter expert) a recipe for disappointment.

But somebody who has spent their time in a corporate environment, learning an area, and then able to translate that into, you know, a startup environment, then I’d encourage them to be entrepreneurs, and entrepreneur owners.

Pacific-Tier: That’s great advice. Give a little pitch for you company. Where do we find you?

Matt Hiles: You can find our company at Mosaic Networx, and the domain is mosaicnetworx.com . if you would like to reach us we’d be happy to hear from you.

Pacific-Tier: Thank you very much for the time!

Here is the entire AUDIO recording of the interview

"The Founder and Chief Executive Officer of Internet Inventions, Bob is a true Silicon Valley entrepreneur, raconteur and original. Bob has combined his technical ingenuity, entrepreneurial ability and team building skills to found successful companies time and again over the past 20 years." (Silicon Valley Business Television)

Bob Evans always has ideas. Ideas to make his work, and the work of others, more useful and efficient, as well as easier. We first met Bob when he asked if it was "OK" he develop a route server for our young Internet Exchange Point (Any2 Exchange), which would allow the small to medium Internet access and content provider community to find Internet peering easier, and help with introductions within the community that would make Internet interconnections a utility – rather than a high priced value-added service. That little utility now drives one of the largest Internet Exchange Points in the world.

We met Bob this week as he was giving a data center tour and professional advice to a delegation from Ramallah, which came to Bob for mentoring based on his extensive background in all things network and Internet.

For the entire audio interview – click HERE

Pacific-Tier: Bob, please tell us a little about yourself, your background, and how you got into this business.

Bob Evans: Well, it was a long time ago, when there was no world wide web. I ran one of the first Veronica Gopher servers. I was teaching grandmothers how to do email, so they could email their kids in college.

At that time colleges had the first ARP connections, TCP/IP started working right, and that was way back when… That’s how I got started.

I’ve got a high school education, dropped out of college, so generally every time I start a business and build it up, sell it to somebody with lots of money and VCs, they (eventually) ask me to leave because I don’t have a degree. Then they usually do quite well. I’ve been doing this a long time.

But when I started the Fiber Internet Center, what happened was, people kind of thought I was crazy, because I had just had a very successful, one of the first cable modem companies in the US, and by starting the Fiber Internet Center/FIC at a time when the "Dot Bomb" occurred everybody thought, you know, "how are you going to make a living doing that?"

The real key was making the deals. Because all of the (network) providers, and the people with fiber and stuff like that, didn’t really have a lot of customers. So, they were willing to listen to me and cut me some deals. So that’s how I got started with the Fiber Internet Center.

Pacific-Tier: Tell us a little about the FIC. What is the vision, or what is the mission of FIC?

Bob Evans: Well. Its primarily,… like you find colo (colocation) facilities that are "carrier neutral,…" I like to look at us as a network company that is carrier neutral. Although we do sell circuits and services like IP transit (Internet network access), it is kind of a necessary evil, because we only have about 8 or 9 other ISPs (Internet Service Providers) currently serving our market.

So they use our network in the market place to service their customers. So we’ll build a network within our network to service other companies.

Pacific-Tier: So the FIC is present in Northern California, the San Francisco, San Jose, the bay area, as well as Los Angeles. What value does the FIC, or a company like the FIC bring to a market like California that is not available through another carrier like Level 3, an XO Communications, an AT&T, or that scale of carrier?

Bob Evans: Well, most of those companies will go ahead and install their circuit in the basement of a building, or in the MPOE (main point of entry), and then only allow you to use their IP (Internet network access).

For example, Level 3 isn’t about to go ahead and market the fact that I could have VLANs in that building like I would have and connect them to ISPs. That’s the real advantage. A customer could buy a circuit from us, then they can buy Internet transit from us too, then they’ll turn around and say "gee, you are over at the PAIX (Palo Alto Internet Exchange), over at places like Market Post Tower, One Wilshire and everything,…"

Then they’ll say "there’s this other company we’re working with, and we’d like a connection direct to their office. They’re using Cogent bandwidth (or something like that), could you make that come in here on another port?"

And we’ll do it. Most (other) companies will say "no." But that’s actually part of our business strategy.

Pacific-Tier: Well, that’s a great utility, for both the enterprises and other carriers within your area. Where does the FIC go from here? What is in the future of FIC?

Bob Evans: Lately it’s become one of the discovery that most of these business now feel a critical need to not have their mail server, or other types of critical databases remain on their campus. Or they want to have a backup of it someplace else. So we’re now creating a service in other data centers where you get a circuit from us, then we’ll give you another port that gives you a layer 2 connection back to your rack or your server in those facilities, or one of our facilities.

This gives you the added benefit of having your own private network to the back side of your server, then it gives you the advantage of maybe having your server connected to other Internet providers in one of those other colo facilities, or maybe you could even carry that Internet back to your place and have two Internet providers.

So I think the advantage is, if you get a circuit from us you get the flexibility.

Pacific-Tier: Well that’s great. I think it is a great utility for the community. The final question is, you’re an entrepreneur, in tough economic times, with lots of graduates hitting the street, what advice do you have for people who are considering starting their own companies, or emerging entrepreneurs?

Bob Evans: Well, that’s a very good question.

During bad economic time I always find that not to discourage me, and I wouldn’t let anybody discourage them either. The reason is because in the midst of chaos, there’s usually always opportunity. And as soon as you explain that opportunity to somebody else who’s got money or can help you, or needs a service that you have, it’s actually easier to make business deals.

Pacific-Tier: That’s great advice.

Thank you very much. It’s a pleasure to visit the FIC again, and thanks for your time.

Bob Evans: You’re welcome – thank you

Check out the entire Innovators and Entrepreneurs Interview Series

This is the fourth article in a series of interviews with Eric Bender, President of Wilshire Connection (WilCon), the largest independent telecom carrier in Los Angeles. In this segment Eric discusses the future of WilCon, including expansion outside of Los Angeles, wireless topics, relations with local utilities, and some great examples of WilCon’s flexibility in delivering telecom solutions to the LA community.

Pacific Tier: Outside of downtown LA , what is your expansion strategy for going to place like El Segundo, Las Vegas, or other cities, parts of the city?

Eric Bender: We’ve leased dark fiber from other carriers to get to other off-net locations such as El Segundo. We connect Equinix on Maple, so we can do lit transport into that facility. We’re working on a plan that would extend from there to the 365 Main location in El Segundo, and then with a second route back to downtown.

We’ve acquired several 10Gbps wavelengths from LA to Las Vegas, and into the Switch NAP facilities, so we can connect with all their switches and facilities, including the Super NAP, which is a 400,000sqft monster of a data center facility. So we do just backhaul or transport to Vegas. We’ve got approximately 10 gigs of traffic back and forth between here and there now.

I don’t see us building on a long haul type arrangement where we would build far outside of downtown LA. We’ll go off-net by structuring deals with existing customers and partners that we have to get fiber and connectivity into other locations. One of the things we are working on is connecting our buildings in LA to Phoenix, there’s a couple of different data centers in Phoenix – there is I/O, (etc).

Then we might go from Phoenix to Las Vegas, and from Las Vegas we already have to LA (which we already have), but also Las Vegas to San Jose/Santa Clara, and then back down to LA. So we basically have a ring from LA to Phoenix, to Las Vegas, San Jose, back to LA.

Pacific Tier: So would you have arrangements with Edison (Southern California Edison), or Burbank Power, or people like that to extend into Hollywood or Burbank or Glendale?

Eric Bender: Well, we do work with Edison and DWP (Los Angeles Department of Water and Power), and have used them to get to further out locations from downtown. The city of Burbank, we’ve not been successful and have not really chased hard to try and make a deal with them. Our initial efforts to work with them were rebuffed without comment – they had no interest in working with us, which surprised me.

I think we can be creative in building, like at 900 N. Alameda which we now have connected with dark fiber, and there is a variety of ways we can do that. We have connection and interconnection agreements AT&T to use their infrastructure so we can theoretically deploy assets throughout the city and state in coordination with AT&T’s network.

Pacific Tier: Would you consider for example going to San Diego using a capacity swap with Edison where they gave you a pair of fibers to San Diego?

Eric Bender: We do that already, well not to San Diego, but we have done a variety of barters, trades, or however you want to call it, with companies like Edison. With Equinix, with DWP I’d like to – but they have no interest and can’t by law (they are a public utility), and can’t really do that. Level 3, we’ve had discussions with them to do the same thing.

Because I’ve got all this infrastructure, whether it’s the conduit, or the fiber, and it’s all there, it’s paid for, there’s no debt, there’s no cost of capital to me – it’s there, so if I need to allocate a couple pairs of fiber going here or there on my network, and I can get something that I can generate revenue off of on somebody else’s network – we’ve done that many, many times and it’s worked out very well with a variety of companies.

Pacific Tier: Expanding a bit more on WilCon’s flexibility…

Eric Bender: We have no rigidity with anything we do. I think that one of the reasons we’ve been successful, and have continued to grow every month, basically where we’ve continued to grow over the previous month is because we’re not rigid and have a straight line approach to things.

I think the advantage of not being a telecom guy by trade or training and education is that we look at things by how do we get the customers the things they need? When a customer needs something my first response is to say "OK," and the second response is to call my technical guys and say "I just committed us to doing this by a certain point in time, make it happen." And they do.

We’re a small company so we don’t have all the various layers of engineering groups and planning groups to say from here it has to pass from point "A," to point "B," to point "C," before you can be told it can be done, or not be done, or how to do it. We just make it happen and get it done.

The story I tell that kind of tied the real estate to the telecom all together well for me is at one of our buildings, 700 Wilshire we had a t4enant in the building. This goes back to about early 1999. We had put conduit in the ground already, so it was probably early 1999, had already connected the conduit to One Wilshire, as well as 700 (Wilshire Blvd), and we had leased space to this tenant.

Part of their requirement was they needed a couple of DS3s between the meet me room at One Wilshire and their premises they had leased at 700. At that point we were only doing conduit deals with our carrier customers. They were going to Level 3 for this DS3, and Level 3 hadn’t really built out their network yet. Because we put Level 3’s first conduit in the ground in LA (it was conduit we’d installed in our trench and gave them, basically), and Level 3 was provisioning this DS3 through PacBell (now AT&T) or someone, and they were re-provisioning (the DS3) through them.

The timing for the tenant was going to be 90 days, or 120 days wait for the tenant to provision this DS3. The tenant came to us and knew we had started Wilshire Connection. So they came to us and said "we really need this, is there anything you can do for us?"

I didn’t know what to do, I mean we just had conduit between the buildings. So the customer said "all you have to do is this. We have the equipment, you’ve got fiber cable already (because we’d already deployed that), you need this piece of equipment, you’ve got this box on this end, that box on that end, and we’ll tell you how to do it and what works well."

Two weeks later we had that DS3 traffic up and running between the buildings. So we got into the transport business. But that was because we were flexible, somebody needed something, (we had the resources), and we got it done.

If we can see there’s a reasonable return on investment and payback on the infrastructure or cost to deploy new equipment, we’ll do it. We got all this MRV DWDM equipment (dense wave division multiplexing) because we had a very limited amount of fiber into the (particular) building, so the only way I could provision this service (lit bandwidth) was with the DWDM gear – so we went out and bought it.

So the equipment was paid for within a year. The customers can come and go – it doesn’t real
ly make any difference (other than we want to keep customers), because it is already paid for. So we’ll continue to do that, and that’s how we’ve continued to grow into the buildings that we have, as well as the transport circuits that we do between all these buildings, as well as connecting into Vegas, and the same to Phoenix, Santa Clara, and other cities.

Pacific Tier: So what is the future of WilCon, where do you go from what you have today?

Eric Bender: several years ago we had some strategy discussions, and (since) we are a dominant carrier in LA, we’re known around the world as the provider of choice, the connector of choice here in LA. We have a great relationship with Equinix, and CoreSite now (CRG), so we get a lot of referral business from those guys. We’ve provided such good service to our customers that big carriers such as TATA, PCCW, and others that they just come straight to us when they have new requirements.

So we had this discussion "let’s go and expand, and into other markets" and we took a very quick and dirty analysis and determined that it costs so much to build out and deploy in these other markets, and we know other companies have been doing similar kinds of things, and we might be providing a similar type of service. In order to do that (expanding into other markets) we’d have to take on an additional equity investor, or venture funding, and then all we’re doing is servicing somebody else’s debt.

So, does that make sense (expanding) or should we continue to be the provider here, and continue to grow which has made us a profitable company. So we said "we are gong to do that, and we’ll expand and provide services outside of LA in various ways that make sense, and does not require a huge capital outlay, and we’ll do some – but we’re not going to spend millions of dollars to deploy."

The reason many companies went bankrupt was because they built these massive networks, and there was no r4evenue to service the debt on them.

So our future is to grow in selected and strategic ways. It will probably be between LA and other cities by usi9ng lit waves, and then establishing a POP (point of presence) in a couple buildings, and then maybe in some cases building a bit in those markets to connect those buildings.

In Dallas we’ve looked at buying an existing network, but there’s really no need to connect between those facilities – there’s no business with that. Chicago was no different. San Francisco, San Jose, Santa Clara are all so spread out its hard to justify a new build or new construction.

I think working with companies like Allied Fiber which is building their long haul network, and working with them to be the local metro piece to what they’re selling to their customers on the long haul – we can tag along with them, working well with them in LA, and possibly duplicate that with them in other markets.

Pacific Tier: Is there any wireless in WilCon’s future?

Eric Bender: I don’t really see that. We did look at partnering with some wireless, WiMAX, or other guys using Terrabeam or some of these free space optics. It seemed like they just had such a long way to go before it would make it worthwhile, and that in LA maybe it’s a problem. Trying to negotiate the line of sight, and rooftop antenna rights and those things,… that we didn’t really get anywhere. It became cost prohibitive I think.

So it would be nice to do something like that. We looked at that (wireless) to support NANOG (North American network Operator’s group) several years ago when they were here. We were in Library Tower (the US Bank Building) on the 56^th floor. I had a line of sight directly from my window to the hotel, but it was too far. We would have needed a couple of hops to be able to deliver a gig (Gigabit connection), and it was a disaster for them to have it there (at that hotel).

They had to go buy a DS3 from AT&T (PacBell). If they had done the conference downtown I could have connected them with no issues.

Pacific Tier: Any final words you would like to say to the global telecom or Internet community, or Los Angeles? Why LA, or with WilCon, downtown LA is an attractive place to do business?

Eric Bender: We’re number one!

I think LA is a great place to do business. From a location perspective its perfect, as its the gateway to the US. You can get anywhere from LA, whether its physical transportation, or from a telecom infrastructure perspective. I think if you come to LA as a carrier, as a provider, as a bandwidth user, in this industry – and you’re going to locate in one of the half a dozen facilities (in downtown LA), whether an Equinix, Level 3’s gateway, One Wilshire, or TELX Carrier center.

Any of the main big facilities here (and Switch and Data) you are going to need, ultimately, to connect to customers that are in one of the other facilities – somehow you’re going to find us. You may be referred to us, do your research – ultimately you are going to find we can provide that connectivity.

We are really easy to do business with, as any of our customers will tell you. And I think that in all the years we’ve been in business, since 1998, I think there are maybe two deals we’ve lost because of price, that we were not the lower cost provider for that particular deal. And, we try to provision within 24~48 hours (of an order).

Pacific Tier: That is outstanding. Words for the global telecom and internet community from Eric Bender, President of Wilshire Connection. Thank you very much.

John Savageau for Pacific Tier Communications, Long Beach

====

This concludes our interview with Eric Bender, President of Wilshire Connection in Los Angeles.  you can contact Eric at ebender@wilcon.com for more information on Wilshire Connection.

The entire interview is available online.

Previous entries in this series include:

• Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
• Part 2 – WilCon Takes to the Streets
• Part 3 – WilCon Manages Infrastructure Risks

This is part three is a series of interviews with Eric Bender, president of Wilshire Connection. Wilshire Connection, or WilCon, is the largest independent local network and neutral fiber infrastructure provider in downtown Los Angeles. In this segment Eric discusses how WilCon managed risk to their network during the initial construction process, the continuing management of critical telecommunications infrastructure, and the role WilCon could play in the event of a major incident impacting the telecom industry in Los Angeles.

Pacific Tier: (on the topic of utility gas and electricity) What risk is there to the infrastructure in downtown LA of an explosion from either electricity or gas, and what would that do to your conduits if it occurred?

Eric Bender: Interesting question… I haven’t really thought much about that, it would, depending on where, you never want to see that. Fortunately with power, the lines typically won’t explode, it is the transformers, which mostly are in the buildings.

The way LADWP (Los Angeles Dept of Water and Power) sets up they bring the high voltage lines into the building, so if the transformer blows up it will be in the building. The transformer is typically not out in the street – but they do have some vaults out in the street, and they have had some explosions, but they have been contained within the vault. We don’t run through any of their vaults so from that perspective we’re OK.

Somebody who is digging and hits a gas line… that’s a different story.

I can say in the roughly 12 years that we’ve had conduit in the ground, and since we were one of the first to dig, I can say that we are lower in the ground, and more protected in many respects, and more of a straight line in routes we’re going without having to jog around any other existing infrastructure that came later.

But we’ve never had one of our conduits damaged or cut. Or interfered with…

The difference between how our conduits or duct banks are typically done vs. many other carriers is that we put in massive amounts of conduit. We use (typically) four inch PVC conduit, and I don’t think we have a single one (trench) which has less than six or eight four inch conduits. Most of them have multiples of that, for example going on Wilshire Blvd between Grand and Figueroa we have an average of conduit in that whole duct bank is probably close to sixty four inch conduits.

Because fifty of them go into One Wilshire, although they kind of peel off in different directions from there, we’ve also got a main ‘48 and they lateral off down into other streets into buildings, so there’s some doubling going on.

But in some areas we have up to sixty four inch conduits so that’s the size of a desk, or bigger, and that duct bank is encased in a concrete slurry around it, surrounded by a couple addition feet of dirt and asphalt on top.

So if anybody digging is going to hit (the duct bank) typically there is warning tape on top of the conduit, concrete, or in the concrete they are going to pull it up. They are going to hit (the tape) or something before they hit the conduit. So unless it is some young, uncontrollable (person) with the backhoe who is on a rampage – no matter what you do you can’t control that.

NOTE: Whenever a company opens a street for utility construction or maintenance you will normally have construction observers and safety observers from not only the company opening the street, but also each company with conduit or utility infrastructure in the immediate area of digging.

But some of the other carriers that put in one or two conduits, they are the ones at risk, like for example an MCI, Verizon, or a QWEST in some cases they typically put in just the two conduits… And you could rip through that even if its encased in concrete before you realize what you are doing.

The conduits are just this big, versus this big (Eric shows a note book size to represent two conduits vs. his desktop to represent WilCon’s conduits).

Pacific Tier: Do you consider yourself the only truly neutral facilities-based carrier in downtown LA?

Eric Bender: I think others consider themselves neutral, but they have other motives as well. I don’t care if I sell dark fiber or lit transport. We can do either, and it doesn’t matter to me which one they want. We have so much fiber, and the infrastructure to continue pulling more and more fiber that I’m never worried about running out of capacity for dark fiber.

A Level 3, or a QWEST, or an XO, they run a network, and they’re obviously not neutral. DWP (LADWP), they’re somewhat neutral because they don’t seem to care whether you take fiber or lit services. I don’t know what they do with lit services or on the network side of things – honestly. We’ve leased from them (LADWP) dark fiber to get access to some off-net buildings, and they’re very easy to work with. They are very rigid, and have no flexibility (LADWP is a public utility), but they’re easy to work with.

We’re probably, as a straight, neutral, not really care who you connect with, we may not be the only one (neutral fiber network), but one of the only ones that would be neutral.

Pacific Tier: One Wilshire is traditionally a center of communications in Los Angeles. Some people think it is a high risk location because there is so much on the 4^th floor and other parts of the building. Some people think that it is meaningless – that it doesn’t have that much value. Do you think the 4^th floor of One Wilshire today is a critical piece of infrastructure, or do you think it is something that is just there, that could be bypassed when and if ever needed?

Eric Bender: I think at this point it would still be considered a critical facility. A lot of carriers and other companies have facilities or locations elsewhere, but because of the way they’ve built their networks from the beginning, One Wilshire has always been the central point for them.

They may not have grown and expanded there, or they may have moved things off to other locations, such as 600 W. 7^th, 818 (W. 7^th ), or even outside of LA, and use a company like us, or some other carrier to make that interconnection or virtual connection between their two facilities. One Wilshire tends to have been their primary facility.

I think that over the years that’s more applicable to legacy carriers, the bigger carriers, the ones that have been around for a long time.

I think the Internet type of carrier that’s either VoIP or an Internet company, content CDNs (content delivery networks), and those – One Wilshire’s not as important to them at all because in my limited technical knowledge it is easier to reroute that traffic to other servers – they have more mirror facilities than a switch would have on the telecom side of things.

Pacific Tier: Is there a business continuity plan, or disaster plan, in the event One Wilshire or another facility like 60 Hudson (New York), or the NAP of the Americas (Miami) anybody has thought about or put on the shelf in the event one of those critical facilities has a catastrophic failure?

Eric Bender: I am not aware of any common, for the
greater good, where all the carriers have participated in developing that, or working out some kind of contingency plan. That’s actually a really good question.

You’d think that after 9/11 where the infrastructure was so significantly damaged that in various other markets such as LA, Chicago, that there would be some kind of a group of organizational effort to have dealt with that. I am not aware of one. They may have one, but I am not aware of it, and they never invited me to participate.

Pacific Tier: Is WilCon positioned, in the case of a worst case scenario in Los Angeles, to assist the community and assist the industry in recovering to an alternate facility if that occurred?

Eric Bender: Sure, I mean, our infrastructure that we built, and that we control and own, is all primarily downtown LA, so… in a worst case One Wilshire becomes untenable, well a lot of our fiber doesn’t all home run into One Wilshire, but a lot of it does go in and turn around, coming back out again.

There would be disruption, but it could be brought out and bypassed. We have diverse paths into most buildings that we connect so we can certainly do it.

Not in LA, but when in the Mediterranean last year when they had the three or four cuts, (several of) our customers were impacted. I sent them an email and said I doubt there is anything we can do here , but if there is anything you need that we can help you with, let me know and we can work with you.

And actually two of the customers said "yeah we need to reroute some connections to put it on a different side of their ring (in the Med and Pacific) that we could do in about five hours with a couple pairs of fiber for them, and they were able to reroute their traffic, or some of their traffic, and lessen the impact of those (submarine fiber) cuts.

Pacific Tier: So WilCon would consider yourselves a very flexible, agile part of a recovery plan, and would not be rigid in your provisioning process, and that you would work with the community to recover from a disaster?

Eric Bender: I agree with that!

=====

This ends the third segment of this series. In the next part, Eric will discuss more of the future of Wilshire Connection, including his visions for expanding WilCon into new markets.

The entire interview is available online.

Previous entries in this series include:

• Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection
• Part 2 – Wiring Los Angeles Part 3 – WilCon Manages Infrastructure Risks

This is the second part of an interview with Eric Bender, President of Wilshire Connection. In this segment Eric talks about the period in 2000 preceding the Democratic Nation Convention, and the aggressive industry build out of conduit, fiber, and telecommunications infrastructure in the downtown Los Angeles area.

Pacific Tier: At what point do you think the city of Los Angeles figured out this would be a really good thing for LA, and it would bring more business and money into downtown?

Eric Bender: I don’t think they ever came to that conclusion. What happened was in 2000 the Democratic Convention was going to be down at the Staples Center in downtown, so I think it was in early 2000 or at the very end of 1999 the city called a meeting and notified all the telecom carrier that had been active or building or doing things that as a part of the preparations for the convention that they would be repaving the streets, resurfacing certain streets, and there was going to be an absolute five year moratorium on any digging or street construction work as a result of this. They wanted the streets to look pretty on TV, and that was fine.

So I don’t think they thought about how much money, or how beneficial the telecom network was going to be to the city, they realized they would need to do something for this convention. The result of that was that it threw somewhat of a panic into the carriers that were in the infancy of building out their networks, and building throughout LA, and it forced a massive infrastructure improvement project, because there was a very limited amount of time before the city was going to shut everything down.

I think the convention was in July or August, and by the end of May everything had to be done. So, we really had about four or five months to build everything and our network probably tripled in size during that period of time. We were one of the only companies that had an active permit that we had pulled for a small segment that we were building that hadn’t started construction yet because it was not that critical to us at the time.

But because of us having an active permit for about a two block, few hundred feet, maybe five or six hundred feet of conduit the city realized they had a major problem that these companies had to be building, had to be constructing – they couldn’t just shut them down for five years. The city did understand that was a problem.

So they said you can build, we will expedite and streamline the process and anyone who has an active permit, well everyone can tag along with that permit. We’ll just change it and build out from there.

So, with our little permit, and the little segment, I think it was on Grand from Wilshire down to 7^th St., and then a little bit more. We basically parlayed that and built it up to 5^th St., then up to Figueroa, and literally tripled our network. So we became the lead builder from that permit on a tremendous amount of other conduit that was built in the city.

Basically they couldn’t keep up with all the activity because there were intersections in downtown LA where there were three backhoes digging for three separate projects. We were doing one, QWEST was doing their own, they didn’t really want to participate much with anyone, and MCI, MFS didn’t want to share conduit so they were doing little things.

And I remember at 7^th and Grand there were three different construction crews and project going on at the same time, and for me it was just a wonderful time because we had this permit, so we had a lot of leeway, and I would walk the streets, stand on the street corner, and these other companies, you know Level 3 and QWEST actually did participate, XO… You know I would say "we’re going down this street to this building," and they’re going "I’m going that way," and we’d shake hands. "I’m taking two conduits in yours, you’re taking a conduit in mine…"

I don’t like to use the term "wild, wild, west," but it was really a very wild and fun time. My best times were just walking around with all the construction going on, with all the lane closures and all the activity. The traffic was backed up, cars were beeping, and would be driving by and I would just be standing there looking at the big hole in the ground with a big smile on my face. It really was the best time.

Pacific Tier: Did you have any catastrophic backhoe cuts or anything disruptive during consutruction? 

Eric Bender: We, during our construction our guys, we never had an incident where we hit any other conduit. There was a water or a sewer line that got hit, and nicked, and that was something that that was not actually on the infrastructure plan. And it wasn’t marked on the street so it wasn’t our problem so much.

There was a close call with a gas line. But nothing happened. But for the most part (Eric knocking on his wooden desk) they did a good job.

The toughest part to build, which we were actually the participant in and XO was the lead builder on 7^th, I think somewhere between Olive and Grand, it shows you the history of LA but, back when they had the street cars, the RED Line, YELLOW Line, and all these other street cars, they never really demoed (demolished) those out.

They just over the years paved over and over and over. As they were excavating and going down the road, the path where they were going, and which later became obvious because of the railroad, they had to rip through all the railroad ties. It was terrible.

Because of that, then they had to get an archaeologist because you know you have bricks from the support for the railroad, because they are historical. I don’t know what they would do with it, it is in the middle of 7^th St (a major road in downtown LA) and Grand Ave, that can’t, you know set up a museum in the middle of it, but it shut the project down. The city had some very peculiar rules that you cannot dig on Sundays that go back to the old blue laws basically, no construction activity on Sundays.

Some streets they won’t let you work during the week, you know it goes back to the days when downtown was a main shopping area, so you know, so you couldn’t dig during the week. You could only dig on Saturdays from nine (a.m.) till three (p.m.) which means you get about from ten till two because you have clean up and everything.

So some projects that should have been done within a week took like eight weeks to get done. So the city tried, but they were not the easiest. The only company that got to work on Sunday was QWEST, because, well, nobody is really sure exactly what happened or how, but they did have several projects where they were working on Sundays.

====

This ends part two of the "Wiring LA" interview series. Part Three will explore some of the risks of building such a dense concentration of telecom projects within the downtown LA area.

The entire interview is available online.

Previous entries in this series include:

Part 1 – Wiring Los Angeles, an Interview with Eric Bender, President of Wilshire Connection

John Savageau, Long Beach

Downtown Los Angeles is among the most densely connected telecommunication hubs in the world. A dozen buildings in LA’s city center house the world’s largest Internet networks, telecommunication carriers, content management networks, and entertainment companies – interconnected through a complex mesh of submarine fiber optic cables, terrestrial cables, and internet exchange points.

With more than 500 networks and carriers operating within the LA city center, one company stands out from the crowd as a leader in bringing the global telecommunications and Internet community together. Eric Bender is president of Wilshire Connection, a facility-based carrier focusing on providing neutral, high capacity fiber optic cable interconnecting the most important buildings in Los Angeles.

Wilshire Connection (WilCon) has a vision to free the LA telecom and Internet-enabled community from the burden of developing a highly meshed inter-building infrastructure, allowing each company to focus on bringing value to their global network interconnections and relationships. Eric Bender, President of Wilshire Connection, is the man behind WilCon’s dramatic success in wiring Los Angeles.

This interview will cover, in a multi-article series, the history of modern telecommunications in downtown Los Angeles, the role Wilcon played in LA’s redevelopment, the period of rapid and chaotic build out prior to the Democratic National Convention in 2000, the risk of high density telecom infrastructure, and the future vision of Eric Bender and Wilcon.

Pacific Tier: Today we have Eric Bender, President of Wilshire Connection. Eric, can you tell us a little about yourself, and how Wilcon came around?

Eric Bender: Sure, we founded Wilcon, Wilshire Connection back in 1998. I am one of the founders and one of the partners, and have run the company pretty much from the beginning. I’m a real estate guy by background. I graduated from college and have always worked in the real estate field, whether its commercial property, management, brokerage, home build, whatever…

Back in late 1996 I started working for a Hong Kong based family which was acquiring some office buildings in downtown LA, one of which was 611 Wilshire Blvd which was directly across the street from One Wilshire. Back at that time telecom deregulation had just been passed, the 1996 Deregulation Act, and there was a boom of CLECs formed and started. Nextlink, the precursor to Level 3 (Kiewit and Sons), MFS was expanding into a variety of things, Worldcom, a variety of these companies were just starting up in that time frame.

For a lot of different reasons One Wilshire had become kind of a focal point for telecom, primarily because it is a 30 story building, had very clear line of sight to the east, and back in the day when microwave transmission was the primary factor, (One Wilshire) had great eastward pointing microwave capabilities. And because of this many of the deregulated companies had got into One Wilshire.

Because our building 611 Wilshire was directly across the street, and only 50% occupied, and One Wilshire was still an office building that only had a meet me room component to it, – that was really it, from a real estate side of things we said "we have all the space, you have all these new companies coming and can’t do it, send them over to us, we’ll fill our building with them, and let’s put some conduit between the buildings together as a virtual extension of (the meet me room) of One Wilshire.

Additionally we put in some conduits that the One Wilshire building actually provided for.

A year or two later we had bought, and were converting other buildings in downtown (700 Wilshire, 818 W. 7^th St.) and saw a need to have those buildings connected back to 611 and One Wilshire.

Standing on the street corner thinking about it we decided that we should become a facilities-based carrier. We went and got our authority from the state to become a carrier, CLEC, and I can’t remember the other authorities we received. Then we decided that’s how we are going to do it. We started digging the streets building the network and connecting the buildings, our own buildings, together.

We decided as a way to fund the build out, because at the time, 1997 and 1998 a lot of buildings had empty space in downtown LA, they all saw what was happening not only with our buildings, but other buildings that had empty space, and saw how telecom was taking massive amounts space. And the reality was landlords wanted telecom companies as tenants because they paid higher rent, they paid for their own improvements to build out their space, they paid for their own utilities, the buildings did not have to provide janitorial, nor provide much in the line of services.

So we created, as part of our model with Wilshire Connection, the ability to connect all these other buildings into our network. This tied them back to One Wilshire and all these other telecom buildings. And that was our methodology of raising funds to build our improvements to our infrastructure.

Pacific Tier: So what role do you believe Wilcon played in the development of downtown LA as a telecom center?

Eric Bender: Well, trying to be as modest as I can, we probably played the primary role in that, because, several buildings are telecom buildings today because we were building our network and including them. They may have become a telecom building in some way differently, but the fact that we connect them into our network which covers and ties all the building together on a neutral basis was really the instrumental part, including 600 W. 7^th St, 800 Hope, 818 W. 7^th St., which has Level 3’s gateway, Equinix, and others – brought these buildings back to One Wilshire and to each other – together they became the success that they are.

Pacific Tier: What was your criteria for selecting the buildings that you POPed (built a point of presence) or entered, or dropped cable?

Eric Bender: We were note really picky. At the time we were building the network it depended on which buildings came to us, what they were willing to contribute to the cost of (build out), and how important it was to them to be connected to the network.

Pacific Tier: The actual process of laying conduit in downtown Los Angeles, did the city support you?

Eric Bender: Not initially, it was quite challenging. Part of it was because a lot of this activity was new and we were building our network, planning our network to cnnect the buildings. But you had the other companies that were building their networks, such as Level 3, MFN (actually they came a bit later), MCI was adding a lot more conduit and buildings, and others. The city was not prepared, I don’t think, from an engineering perspective and plan check to deal with all that, so they were backed up, and you had companies like Level 3 which was building a network throughout the entire city basically.

The city’s engineering department that reviewed the plans – you would go in and the entire office would be stacked four, five , six feet high with plans of what they were planning to build. And it took months and months, 6 to 8 months at times.

This is the end of part 1 of our interview with Eric Bender. Additional segments to the interview will discuss the actual buildout of Wilscon’s network prior to the Democratic National Convention in 2000, the risk of having such high density infrastructure in downto
wn Los Angeles, and future plans for Wilcon and Eric Bender.

====

John Savageau, Long Beach

You can listen to the entire interview online at Pacific Tier

Check out the entire Innovators and Entrepreneurs Interview Series

In 2005 Marty Plotnick, a long time supporter to information technology in Honolulu, put together a brief history of the telephone company in Hawaii.  History should never be lost or forgotten, so here a reprint of Marty’s great contribution.

Select the link next to the short entry and you will be directed to Marty’s old CyberZone website for the full page.

Marty is now retired, living in Honolulu.  If you would like to contact him, please send a note to info@pacific-tier.com  and we’ll get the message through.

A History of the Telephone Company in Hawaii

PART 1.  In earlly 1878, Maui’s Charles H. Dickey installed Hawai’i’s first two telephones between his home and his store. The phones were rented from a Mainland firm and ran on wet cell batteries.On December 23, 1880, a charter was granted to the Hawaiian Bell Telephone Company.  (Bell had nothing to do with the company. The name "Bell" was added to honor Alexander Graham Bell.) The primary investor was the Orient Bell Telephone Company of London and the first order was for 50 phones from a Belgium manufacturer. 

PART 2. For most of 1900 transmitting/receiving stations were set up, but failed for the most part. The Marconi techies were sent back to the Mainland and Marconi sent an "expert" to solve the problem. He did. He used Ben Franklin’s kite experiment and found the right locations for the stations. Service was spotty and Wireless Telegraph ran into debt. The "expert" bought the judgments for $600. and got local businessmen and sugar planters to put up $1,000. a month to continue attempts to get reliable service.

PART 3. The first strike against Mutual was called by the IBEW, beginning on May 3, 1920. Approximately 75 of Mutual’s employees were union members, including 60 linemen.  During negotiations before the Public Utilities Commission, Mutual surprised everyone by voluntarily offering a general wage increase totaling $87,000 a year if the PUC would approve it.

PART 4. The Great Depression had begun in October 1929 and by late 1931 was beginning to bite at Mutual’s business. During late 1931, 180 small businesses disconnected their phone service but a whopping 823 residence subscribers cancelled. In the country districts, six percent cancelled along with the nine percent on the city side.

PART 5. Some of these wait-listed folks had been on the list for up to four years. But even though had laid 12,500 miles of cable in 1945, it still had problems getting copper and lead, as government restrictions had not been lifted. It took over six months from War’s-end to get 200 lines installed. From that point in March 1946, 500 went hot in May, 800 in June, etc. By the end of 1940, 3,800 new lines were installed. 1946 began with 978 employees, double that of 1940. The payroll went from 1940’s over $1 million to $2.25 million.

PART 6. In August, 1959, Statehood became reality after decades of attempts. At Hawaiian Telephone–the new name for Mutual Telephone–growth was spurred by the Territory’s new status.  By the end of 1960, Hawaiian Telephone had 200,000 lines installed.  These customers now had dial service assistance and toll-free dialing on all islands was operational. This feat made the Island of Hawaii the single largest toll-free calling area in the United States.

PART 7.  This installment begins with the end of the company’s first century and beginning of it second in 1983. This benchmark was also the start of an upheaval in the telecommunications industry unprecedented in the entire history since Watson answered Bell’s hail.   This new era exploded the sedentary legacy phone company attitudes nationwide with a whole new vocabulary of technology and business concepts.

PART 8.  In 1989, after 105 years of service, Hawaiian Tel went modern by leasing an IBM 3090-200E mainframe computer to handle its accounting "chores". The company would not disclose the cost of the lease.

PART 9.  1991 saw HawTel finishing its 1990 fiscal year ending with $539.3 million in gross income and $52.0 million in profit. The company paid its parent–GTE Corp.–a record $$43.5 million in dividends.

John Savageau, Long Beach