Welcome to Rise Forums

Join our fantastic community to connect with like-minded website owners, WordPress users, and online entrepreneurs.

Downtime Tonight Due to Damage to Host's Fiber-Optic Cable

Discussion in 'Community Announcements' started by Kevin Muldoon, Nov 2, 2015.

Thread Status:
Not open for further replies.
  1. Hi everyone.

    Apologies for the downtime tonight.

    Around five hours ago a fibre-optic cable between BHS and Newark, NJ, was down. This caused all of my websites, and tens of thousands of others, to go offline.

    I hate downtime as much as the next person, however it is difficult for me to be angry at my hosting company for this.

    Physical problems can occur in the hosting game and they were quick to keep us all informed of it and resolve the situation.

    Anyways, we should hopefully not experience any more downtime.

    Kevin
     
  2. I guess we'll have to forgive you this time.

    Just kidding. I didn't realize it was down, and yes, things happen. It's probably a wonder everything works as well as it does.
     
    Kevin Muldoon likes this.
  3. Kudos to my hosting company OVH. I experienced downtime the other day, but they emailed tonight to explain what happened.

    They have taken big steps to ensure it can never happen again and ensured that all customers will be compensated for the downtime they experienced.

    Fair play to them.

    Hello,

    On Monday November 2, 2015, the BHS datacenter suffered a new network problem impacting all BHS Datacenter customers. This incident was similar to 5 months ago, when 3 pairs of our optical fibers were cut between Beauharnois and Montreal on the North route.

    Two years ago, we began adding redundancy on the 103 KMs of fiber on the South loop. Since then, there have been long delays concerning this project, bringing the time of completion to 24 months instead of the planned 12. Work was finished this week and only in response to the urgency of the situation. We forced the installation of the new South fiber optic line, thereby reducing the duration of downtime of the public network from 10 hours and 51 minutes to 4 hours and 55 minutes.

    Now, the datacenter is connected in total redundancy by the North loop and the South loop. We have 5 directions: 2 towards New York, 1 towards Chicago and 2 towards Montreal.

    Three years after starting the project, we are continuing the work on improving the 2nd route towards New York where all that remains is to solder the optic fiber at the Canada/USA border. Negotiations are underway to connect Toronto and Detroit, MI.

    Here is what our dark fiber and DWDM network connecting BHS looks like:

    We own:
    - 4x10 Gbps between BHS and Montreal via the North loop
    - 4x10 Gbps between BHS and Montreal via the South loop
    - 6x100 Gbps towards New York/Newark via the South loop
    We rent:
    - 10x10 Gbps towards Chicago via South loop
    - 20x10 Gbps towards Newark via the North loop
    - 4x10 Gbps towards London (through Canada) via South loop
    - 4x10 Gbps towards Paris (through USA) via the North loop


    [​IMG]


    Within 3 to 4 weeks, we will finish the work on our own 2nd route towards New York/Newark and we are going to activate an optical failover of 6x100 Gbps through the following 2 routes: BHS/NYC-NWK. This will allow us to terminate the rental of the 20x10 Gbps through Newark.


    [​IMG]


    We are going to increase the capacity of Montreal/Chicago to 2x100 Gbps then 4x100 Gbps while waiting for our own route to be finished between BHS and Chicago by Toronto/Detroit. We cannot yet provide an ETA, knowing that we lack 800 KMs between Toronto and Detroit, MI. We already have in place BHS towards Toronto and Detroit, MI towards Chicago, IL.


    [​IMG]

    At the same time, we are working on the network in the USA between New York and Newark (NY) then Chicago (IL) and Ashburn (VA). This new USA network is planned to be deployed in Q2 2016 and will not only provide redundancy from BHS towards Chicago through New York but also allow for the connection of a datacenter in the USA.

    We sincerely apologize for the latest BHS downtime. We underestimated the time and the complexity of the fiber optic network construction projects in Canada, thus creating an unreasonable delay in securing each kilometer connecting the BHS Datacenter. Even though today it's done and we have 100% redundancy, we are really sad to announce this under such circumstance.

    Within the next 10 days, we will offer compensation which conforms to our SLA for the month of November.

    The exact times concerning the incident:
    - down at 11:53 AM GMT-5 (5:53 PM GMT+1)
    - at 4:48 PM GMT-5 (10:48 PM GMT+1), traffic to the BHS datacenter from the outside returned to normal thanks to the optical fiber by the South
    - in contrast, the links between BHS and the other OVH datacenters and the VAC (Anti-DDoS) were affected until the complete repair of the North fiber. Traffic was able to resume at 10:44PM (04h44 GMT+1).

    Best Regards,
    Octave
     
    Heather likes this.
  4. Impressive. They seem like a rare breed... so open, honest, and communicative. That's really great.
     
Thread Status:
Not open for further replies.

Share This Page