PLANNED NETWORK OUTAGE TUESDAY 02-27-2024 NOON - 2:00 PM

We received word from Comcast that they will be performing maintenance on their Internet "pipeline" this Tuesday from noon to 2:00 pm, and that our circuit "may" be affected. Our Internet runs through their "pipeline" from Sacramento to Vacaville.

What that means is that there may be NO INTERNET during that maintenance window. Please make other arrangements (e.g. cellular hot spot) if Internet is required during the downtime.

ALL towers and relays may be affected.

BERRYESSA PEAK SPEED ISSUE

Techs are on their way to Mt. Vaca and Berryessa Peak tower to realign the primary backhaul. The backhaul is the main Internet feed between Solano and Yolo Counties. The backhaul blew out of alignment at 1:00 am this morning. Yolo County customers are currently on the secondary (backup) backhaul which cannot pass as much traffic, so things will move a bit slower until the primary backhaul is realigned.

No other towers or relays are affected.

MCQUEEN AND BERRYESSA TOWERS ON SECONDARY BACKHAUL

Wednesday evening around 5 pm the primary backhaul (main Internet supply line) failed. The secondary backhaul immediately kicked-in and is currently operational. Techs will investigate the issue at daylight. The secondary backhaul does not provide as much speed as the primary so there may be some slowness. Please refrain from streaming videos/TV or gaming until we can resolve the issue.

No other towers or relays are affected.

PLANNED NETWORK MAINTENANCE - COGENT COMMUNICATIONS

We received a notice from our upstream Internet provider (Cogent Communications) that they will be performing network maintenance at their Sacramento network center starting around midnight on 11-22-2023 and ending by 5 am 11-22-2023. During this maintenance window, we will experience one or more brief interruptions in service while Cogent completes their maintenance activities; the interruptions are expected to last less than 15 minutes (each circuit migrated separately) total. However, due to the complexity of the work, your downtime may be longer. Customers may also see some reconvergence during the maintenance window.

INVESTIGATING A WIDE-SPREAD INTERNET OUTAGE

Investigating a wide-spread Internet outage. Appears to be upstream from us.

UPDATE 2:00 pm: it does appear to be an issue at Comcast's end. We're contacting them now to see if they have any information. No ETA.

UPDATE 5:45 pm: Comcast has found a break in the fiber line in Vacaville and is working to repair it. Still no ETA but getting closer.

FINAL UPDATE 10-19-2023: The fiber line break was on Peabody Rd (Vacaville) and was caused by new contruction work. Our Internet "backbone" (the source of our Internet) comes from a company called Cogent Communications. Their Internet travels through "underground pipes" (from Canada to Vacaville) via "strands of hair" (called fiber cable). Well, someone broke through the pipe and fiber at a Peabody Road construction site Wednesday around 2 pm and didn't know it. Although Cogent owns the fiber, Comcast owns the pipes. Comcast techs found and repaired the damage around 6:35 pm.

NETWORK OUTAGE

There's a network outage affecting all customers. PG&E cut power to the main tower at 9:00 am today and we switched over to battery backup power automatically. However, the circuit breakers keep popping. Techs are onsite trouble-shooting. No ETA.

NETWORK OUTAGE AFFECTING SOME CUSTOMERS

Looks like one of the switches locked up at the main Lagoon Valley Tower. A tech is enroute to reboot it. A remote reboot did not work. ETA 9 am.

UPDATE 9:00 am: turns out it's a bad fiber cable coming into the switch. Techs are running a new cable now. ETA 11:00 am

VACAVILLE PG&E OUTAGE AT LAGOON VALLEY NEIGHBORHOOD

PG&E outage for 500-5000 customers. Our main tower is on battery backup for ~6 hours before the network goes dark. Hoping PG&E has power restored before then.

UPDATE 7:30 am: PG&E power restored at Lagoon Valley tower.

UPDATE 07-05-2023 midnight: looks like PG&E is down again. NETWORK IS DOWN FOR ALL CUSTOMERS.

UPDATE 07-05-2023: a PG&E outage at midnight has alledgedly been fixed (PG&E website), however, we still have no building power at the Lagoon Valley site (all network traffic flows through Lagoon Valley). The PG&E site says the surrounding neighborhood is without power and I suspect they don't have our building correctly listed. I will have a tech drive up to check on building power as soon as possible this morning.

Currently no ETA on when the network will be back up. ALL TOWERS AND RELAYS ARE AFFECTED.

TROUBLESHOOTING AN ISSUE AT BERRYESSA PEAK TOWER

Seems to be a hardware issue at Berryesa Peak Tower. Investigating. No other towers or relays are affected.

UPDATE 8:30am: it looks like either the McQueen Tower side or the Berryessa Peak Tower side backhaul (main Internet feed) has blown out of alignment. Techs are on their way.