Techs will be onsite at English Hills Relay this morning to trouble-shoot the network hardware. We replaced a router and switch last week because the transmitters would spontaneously reboot throughout the day. This weekend the problem resurfaced. There may be intermittent outages and we test various electrical components associated with the solar equipment.
No other towers or relays are affected.
We are repairing some water damage to the equipment box at the English Hills Relay. The work should not take more than 15-20 minutes.
No other towers or relays are affected.
The switch appears to have failed. A tech is enroute to trouble-shoot. Current ETA is 4 pm.
No other towers or relays are affected.
We received word from Comcast that they will be performing maintenance on their Internet "pipeline" this Tuesday from noon to 2:00 pm, and that our circuit "may" be affected. Our Internet runs through their "pipeline" from Sacramento to Vacaville.
What that means is that there may be NO INTERNET during that maintenance window. Please make other arrangements (e.g. cellular hot spot) if Internet is required during the downtime.
ALL towers and relays may be affected.
We received a notice from our upstream Internet provider (Cogent Communications) that they will be performing network maintenance at their Sacramento network center starting around midnight on 11-22-2023 and ending by 5 am 11-22-2023. During this maintenance window, we will experience one or more brief interruptions in service while Cogent completes their maintenance activities; the interruptions are expected to last less than 15 minutes (each circuit migrated separately) total. However, due to the complexity of the work, your downtime may be longer. Customers may also see some reconvergence during the maintenance window.
Investigating a wide-spread Internet outage. Appears to be upstream from us.
UPDATE 2:00 pm: it does appear to be an issue at Comcast's end. We're contacting them now to see if they have any information. No ETA.
UPDATE 5:45 pm: Comcast has found a break in the fiber line in Vacaville and is working to repair it. Still no ETA but getting closer.
FINAL UPDATE 10-19-2023: The fiber line break was on Peabody Rd (Vacaville) and was caused by new contruction work. Our Internet "backbone" (the source of our Internet) comes from a company called Cogent Communications. Their Internet travels through "underground pipes" (from Canada to Vacaville) via "strands of hair" (called fiber cable). Well, someone broke through the pipe and fiber at a Peabody Road construction site Wednesday around 2 pm and didn't know it. Although Cogent owns the fiber, Comcast owns the pipes. Comcast techs found and repaired the damage around 6:35 pm.
There's a network outage affecting all customers. PG&E cut power to the main tower at 9:00 am today and we switched over to battery backup power automatically. However, the circuit breakers keep popping. Techs are onsite trouble-shooting. No ETA.
Looks like one of the switches locked up at the main Lagoon Valley Tower. A tech is enroute to reboot it. A remote reboot did not work. ETA 9 am.
UPDATE 9:00 am: turns out it's a bad fiber cable coming into the switch. Techs are running a new cable now. ETA 11:00 am
PG&E outage for 500-5000 customers. Our main tower is on battery backup for ~6 hours before the network goes dark. Hoping PG&E has power restored before then.
UPDATE 7:30 am: PG&E power restored at Lagoon Valley tower.
UPDATE 07-05-2023 midnight: looks like PG&E is down again. NETWORK IS DOWN FOR ALL CUSTOMERS.
UPDATE 07-05-2023: a PG&E outage at midnight has alledgedly been fixed (PG&E website), however, we still have no building power at the Lagoon Valley site (all network traffic flows through Lagoon Valley). The PG&E site says the surrounding neighborhood is without power and I suspect they don't have our building correctly listed. I will have a tech drive up to check on building power as soon as possible this morning.
Currently no ETA on when the network will be back up. ALL TOWERS AND RELAYS ARE AFFECTED.
We will be upgrading some hardware at the English Hills relay this morning (~8 am). There will be a brief outage (<15 minutes) as we replace the hardware.
No other towers or relays will be affected.