PLANNED NETWORK OUTAGE TUESDAY 02-27-2024 NOON - 2:00 PM

We received word from Comcast that they will be performing maintenance on their Internet "pipeline" this Tuesday from noon to 2:00 pm, and that our circuit "may" be affected. Our Internet runs through their "pipeline" from Sacramento to Vacaville.

What that means is that there may be NO INTERNET during that maintenance window. Please make other arrangements (e.g. cellular hot spot) if Internet is required during the downtime.

ALL towers and relays may be affected.

MCQUEEN AND BERRYESSA TOWERS ON SECONDARY BACKHAUL

Wednesday evening around 5 pm the primary backhaul (main Internet supply line) failed. The secondary backhaul immediately kicked-in and is currently operational. Techs will investigate the issue at daylight. The secondary backhaul does not provide as much speed as the primary so there may be some slowness. Please refrain from streaming videos/TV or gaming until we can resolve the issue.

No other towers or relays are affected.

PLANNED NETWORK MAINTENANCE - COGENT COMMUNICATIONS

We received a notice from our upstream Internet provider (Cogent Communications) that they will be performing network maintenance at their Sacramento network center starting around midnight on 11-22-2023 and ending by 5 am 11-22-2023. During this maintenance window, we will experience one or more brief interruptions in service while Cogent completes their maintenance activities; the interruptions are expected to last less than 15 minutes (each circuit migrated separately) total. However, due to the complexity of the work, your downtime may be longer. Customers may also see some reconvergence during the maintenance window.

MCQUEEN TOWER DOWN

Looks like the switch at the McQueen tower failed at about 6:45 pm Wednesday evening. A tech will drive up Thursday morning to replace the switch.

No other towers/relays are affected.

INVESTIGATING A WIDE-SPREAD INTERNET OUTAGE

Investigating a wide-spread Internet outage. Appears to be upstream from us.

UPDATE 2:00 pm: it does appear to be an issue at Comcast's end. We're contacting them now to see if they have any information. No ETA.

UPDATE 5:45 pm: Comcast has found a break in the fiber line in Vacaville and is working to repair it. Still no ETA but getting closer.

FINAL UPDATE 10-19-2023: The fiber line break was on Peabody Rd (Vacaville) and was caused by new contruction work. Our Internet "backbone" (the source of our Internet) comes from a company called Cogent Communications. Their Internet travels through "underground pipes" (from Canada to Vacaville) via "strands of hair" (called fiber cable). Well, someone broke through the pipe and fiber at a Peabody Road construction site Wednesday around 2 pm and didn't know it. Although Cogent owns the fiber, Comcast owns the pipes. Comcast techs found and repaired the damage around 6:35 pm.

NETWORK OUTAGE

There's a network outage affecting all customers. PG&E cut power to the main tower at 9:00 am today and we switched over to battery backup power automatically. However, the circuit breakers keep popping. Techs are onsite trouble-shooting. No ETA.

REPLACING THE NE TRANSMITTER ON MCQUEEN TOWER THIS MORNING

We will be replacing the NE transmitter on the McQueen Tower this morning sometime between 7:00-10:00am. There will be a brief outage for only those customers connected through that transmitter (<30 minutes). There are 9 transmitters on McQuuen but only the NE one is affected.

No other towers or relays will be affected.

NETWORK OUTAGE AFFECTING SOME CUSTOMERS

Looks like one of the switches locked up at the main Lagoon Valley Tower. A tech is enroute to reboot it. A remote reboot did not work. ETA 9 am.

UPDATE 9:00 am: turns out it's a bad fiber cable coming into the switch. Techs are running a new cable now. ETA 11:00 am

VACAVILLE PG&E OUTAGE AT LAGOON VALLEY NEIGHBORHOOD

PG&E outage for 500-5000 customers. Our main tower is on battery backup for ~6 hours before the network goes dark. Hoping PG&E has power restored before then.

UPDATE 7:30 am: PG&E power restored at Lagoon Valley tower.

UPDATE 07-05-2023 midnight: looks like PG&E is down again. NETWORK IS DOWN FOR ALL CUSTOMERS.

UPDATE 07-05-2023: a PG&E outage at midnight has alledgedly been fixed (PG&E website), however, we still have no building power at the Lagoon Valley site (all network traffic flows through Lagoon Valley). The PG&E site says the surrounding neighborhood is without power and I suspect they don't have our building correctly listed. I will have a tech drive up to check on building power as soon as possible this morning.

Currently no ETA on when the network will be back up. ALL TOWERS AND RELAYS ARE AFFECTED.