ATT Network

[John Kloehr]

Separate from the bombing, just practical info related to the network infrastructure.

 

The bomb did damage the building.

 

At this time, recovery crews have limited access to the site.

 

When the bomb went off, power to the site was lost.

 

The diesel generators (apparently) started up. Due to limited access, they were not refueled.

 

Some hours after the event, the generators presumably ran out of fuel. This is when outages for various service may have kicked in. Outages ere not on the various monitoring sites in the first few hours after the event, they came in later.

 

There was also a fire in the building, which was apparently suppressed with on site equipment.

 

Crews had limited access yesterday to try to bring power back on site, but had to evacuate when a previous fire restarted overnight.

 

911, hospital services, and other elements of infrastructure affected. Anecdotal reports of NICS not accessible. Nashville Airport was "down" for a few hours but is operational again.

 

I have observed certain sites are not accessible over Comcast (main one being power outage information for my county) that is accessible if I use my Verizon phone as a hot spot. So Comcast must route some traffic through ATT via Atlanta (leaving out details, reporting what I can see and from various reports).

 

Atlanta is also seeing problems related to the ATT network

 

ATT has added a couple mobile cell "towers" in Nashville, but ATT phones may have problems elsewhere.

 

Various regional stores (Walmart, etc.) can not take payments by card, cash only. Other ATM networks and payment sites (some gas stations?) may be affected.

 

What are you seeing?

[Tennessee williams]

14 minutes ago, John Kloehr said:

 Anecdotal reports of NICS not accessible. 

It's not anecdotal here. Two local gun stores corroborated, it is down for middle tn area. I think Widder said it was down over in East TN too.

My Verizon has been spotty today. It wasn't yesterday. My mother's phone is still not working today. It's ATandT.

[Forty Rod SASS 3935]

My local radio station reported that the police believe this to have been "a deliberate bombing."

 

No s---, Dick Tracy!

[Blackwater 53393]

I’m bootlegging off of my son’s wifi, so I have limited access to internet and e-mail. No text or phone service otherwise. 

 

In the last hour or so, I have seen the ATT come up on the phone with no bars.  It drops back to no service after a moment.  I have not attempted to reset my modem in the last few hours. I will try that shortly.

 

As time passed yesterday morning, more and more services began to fail . At one point, all four of the Nashville local TV stations went off the air for a short time. They quickly returned to the air.  My satellite TV service has not been interrupted.

[Ace_of_Hearts]

All the providers are changing their tower over from 3G to 4G/5G.

Service from ALL providers on 3G will come to an end on December 31, 2021. Some providers will cease on the lower bands before that.

When they are doing the changeover service coverage will be spotty or non-existent.

 

https://www.multitech.com/documents/publications/marketing-guides/MT_Anticipated_Sunset_Cellular_Carriers_PDF_2020-06-23.pdf

 

 

[J.D. Daily]

It seems strange that the AT&T central office would have onsite fuel storage for less than 24 hours of operation if not days of fuel capacity in a part of the country that is subject to sever weather events that can take out utility power for days.  Also the media are describing the facility as a historic building.  This means the building originally had electro-mechanical switching equipment & space for operators.  These CO's required heating; so, besides fuel for diesel generators they had back-up fuel for the boilers.  So, these old CO's have large fuel tanks; way more capacity than required to run the emergency generators for weeks.  It is more likely that it was damage to the building including major damage to their Emergency Power Supply System.  The fact that service hasn't already been restored to the whole area affected will mean that AT&T will assemble a large committee to study the event & initiate infrastructure upgrades to limit disruptions should scat hit the fan in the future. 

[John Kloehr]

6 hours ago, J.D. Daily said:

It seems strange that the AT&T central office would have onsite fuel storage for less than 24 hours of operation if not days of fuel capacity in a part of the country that is subject to sever weather events that can take out utility power for days.  Also the media are describing the facility as a historic building.  This means the building originally had electro-mechanical switching equipment & space for operators.  These CO's required heating; so, besides fuel for diesel generators they had back-up fuel for the boilers.  So, these old CO's have large fuel tanks; way more capacity than required to run the emergency generators for weeks.  It is more likely that it was damage to the building including major damage to their Emergency Power Supply System.  The fact that service hasn't already been restored to the whole area affected will mean that AT&T will assemble a large committee to study the event & initiate infrastructure upgrades to limit disruptions should scat hit the fan in the future. 

Looks like they did run out of fuel, and then the batteries the generators charge ran out of juice. The batteries ran too low, some have reversed polarity and may be permanently damaged.

 

This page from SardisTel posted info from ATT updates:

 

https://www.sardistel.com/announcements/

 

Please note the top of the page states all services are restored; this applies only to SardisTel customers, not all ATT customers. Read lower on the page for ATT info.

[Blackwater 53393]

The AT$T building is not a historic building. It IS built in an historic district.

 

As to the power outage situation, the area seldom suffers any prolonged major outages due to weather.  Small sections of the grid may be down for a day or two on rare occasions, but the local utilities are very fast with responses.

 

The only prolonged major failure was over 25 years ago when we had a truly fluke ice storm. That was a once in 100 year event!!

 

The situation here at the moment is that I get sporadic service indicators with no or one bar for as much as a couple of minutes and the LEDs on my modem reflect that something is being done on the system.

[bgavin]

It appears a bomb has been plaguing AT&T in my area for the last 27 years.
When the AT&T tech came to my house (again) for the last time, then lied straight to my face, I kicked their sorry ass to the curb.

[Eyesa Horg]

8 hours ago, J.D. Daily said:

It seems strange that the AT&T central office would have onsite fuel storage for less than 24 hours of operation if not days of fuel capacity in a part of the country that is subject to sever weather events that can take out utility power for days.  Also the media are describing the facility as a historic building.  This means the building originally had electro-mechanical switching equipment & space for operators.  These CO's required heating; so, besides fuel for diesel generators they had back-up fuel for the boilers.  So, these old CO's have large fuel tanks; way more capacity than required to run the emergency generators for weeks.  It is more likely that it was damage to the building including major damage to their Emergency Power Supply System.  The fact that service hasn't already been restored to the whole area affected will mean that AT&T will assemble a large committee to study the event & initiate infrastructure upgrades to limit disruptions should scat hit the fan in the future. 

I agree, my knowledge of COs is the same. We had very large underground tanks at our COs for fuel. Heat & generator separated. I think they could run for weeks on those thousands of gallons. Even the small sites held several hundred gallons.

[Blackwater 53393]

We got SOME services back a little while ago.  Land line and home computer/DSL/Fiber are working now.  Schoolmarm's cell phone is working again.  

[J.D. Daily]

On 12/27/2020 at 7:44 AM, Eyesa Horg said:

I agree, my knowledge of COs is the same. We had very large underground tanks at our COs for fuel. Heat & generator separated. I think they could run for weeks on those thousands of gallons. Even the small sites held several hundred gallons.

When AT&T phased in the Western Electric solid state switches the heating demand was eliminated & the buildings required air conditioning.  The solid state switches reject a lot of heat to the air.  The solid state switches were eventually replaced with more efficient digital switches.  One of the unique features of CO's in larger cities is they are designed so that additional stories can be added easily in the future.  The antitrust suit& technology ended the need for more space as population expanded.  My experience with CO's began in late 80's as an employee of a company that supplies diesel gensets & related equipment.

[J.D. Daily]

On 12/27/2020 at 5:37 AM, John Kloehr said:

Looks like they did run out of fuel, and then the batteries the generators charge ran out of juice. The batteries ran too low, some have reversed polarity and may be permanently damaged.

 

This page from SardisTel posted info from ATT updates:

 

https://www.sardistel.com/announcements/

 

Please note the top of the page states all services are restored; this applies only to SardisTel customers, not all ATT customers. Read lower on the page for ATT info.

The explosion must have damaged the fuel supply system.  This is what happened at several CO's in LA area during the Northridge EQ.  After that they changed their design specifications for emergency generator systems to minimize possible disruptions to engine fuel supply systems.