NorthPoint
IT Field Guide
Network Outage Response
IT Field Guide / Network

Network Outage Response

Runbook Network Critical Updated March 2026 ~5 min read
If residents are at risk or clinical systems are down: Escalate to Anthony immediately — do not wait to complete all steps first.
1
Confirm it's a real outage — not a single device
Takes 2 minutes · Do this before touching any hardware
  • Try a second device on the same network — phone, tablet, or another workstation. If only one device is down, this is a device issue, not an outage. → If single device: restart it, check NIC, re-run NinjaOne agent. Stop here.
  • Test two separate things: a website (google.com) and a ping to an internal IP. This tells you whether it's internet only vs. full network failure.
  • Ask the nurses station if PCC and phones are also affected. Loss of phones + internet = likely ISP or core switch, not a single AP.
  • Check NinjaOne from your cell phone (hotspot or LTE) — look for the facility's devices going offline in bulk. Bulk offline = infrastructure. One or two devices = endpoint issue.
2
Read the hardware status lights
Walk the IDF/MDF — lights tell you where the break is
Ubiquiti (UniFi)
Solid white — normal
Solid blue — isolated mode
Amber — firmware issue
Pulsing red — hardware fault
Off — no power / PoE issue
Cisco Meraki MX
Solid green — normal
Amber — WAN issue
Solid red — fault
Off — no power
Managed Switches
Port LEDs blinking = traffic
Amber = 100Mbps link
Off = no link on port
Key question: Is the WAN port on the firewall showing a link? If the WAN light is off or amber, the ISP circuit is down — skip to Step 4. If WAN is up but nothing works internally, the issue is local.
3
Check the dashboards
Use your phone hotspot if the facility network is down
  • NinjaOne — check the facility's device list for mass offline status. Look at "Last Seen" timestamps to narrow down when it started.
  • Meraki Dashboard (dashboard.meraki.com) — MX appliance status. A red WAN indicator here confirms ISP circuit loss. Check the uplink health graph for the exact drop time.
  • Ubiquiti UniFi console — verify controller connectivity. If the controller itself is at the facility and the network is down, you may need to access it locally via its IP on a laptop plugged in via ethernet.
  • Ubiquiti Protect — if cameras went offline at the same time as everything else, confirms it's a full network event, not an application layer problem.
4
Diagnose and act
Match your findings to the right response
WAN down → ISP circuit loss. Go to Step 5 (ISP escalation). Nothing you do locally will fix this — don't waste time cycling hardware first.
WAN up, LAN down → Power cycle the core switch (graceful shutdown if possible, hard reboot if not). Wait 90 seconds for full boot. If no change, check if a bad cable or newly-added device is dragging down a VLAN.
Partial outage → One wing or floor affected — suspect a PoE switch port failure, AP reboot loop, or VLAN misconfiguration. Isolate by unplugging downstream switches one at a time. Check for broadcast storms in switch logs.
PCC down only → This is not a network outage — this is a clinical system issue. Confirm other internet access is working. Contact Rogi Poblete and open a PCC support ticket directly.
Power cycle helped → Document what was cycled and when in the incident log. Monitor for recurrence. If same switch fails again within 30 days, flag for replacement.
5
ISP escalation
For confirmed WAN / circuit loss only
  • Before calling: note the facility name, service address, and account number (see Lockton/ISP contract in ShareFile or ask Geremia).
  • Tell them: "We have a complete loss of service at [address]. WAN link is down as of [time]. We need an emergency ticket opened and an ETA."
  • Get a ticket number — do not hang up without one. Text it to Anthony.
  • Ask for escalation to a field tech if ETA is more than 2 hours and clinical systems are affected.
  • Document: time of call, rep name, ticket number, stated ETA.
No ISP info on hand? Check the Meraki dashboard — the WAN IP and ISP are shown under Appliance → Uplink. Account numbers are in ShareFile under the facility's contract folder.
6
Temporary mitigation
Keep clinical operations running while waiting for ISP
  • Mobile hotspot bridge: Connect a company hotspot (or your phone) to the facility's firewall WAN failover port if available. Confirm with Anthony before doing this — some sites have Meraki cellular failover already configured.
  • PCC offline mode: Notify the DON that PCC may have limited functionality. Clinical staff should follow paper downtime procedures until internet is restored. Downtime binders should be at the nurses station.
  • VOIP phones: If phones are also down (SIP over WAN), notify the front desk. Route calls to facility admin's cell as a temporary number until restored.
  • Cameras: Ubiquiti Protect records locally to the NVR/UNVR even without internet. Footage is not lost during an outage.
7
Escalation contacts
When to call who
IT Director (Primary)
Anthony Trujillo
Escalate if: outage exceeds 30 min, clinical systems are down, or ISP is unresponsive.
Available via Google Chat or phone.
Network / Infrastructure
Geremia Doan
Escalate if: Meraki or Ubiquiti configuration change is needed, or Anthony is unreachable.
Clinical Systems (PCC)
Rogi Poblete
Contact if PCC is down or inaccessible after internet is restored — may be a session/credential issue.
Facility Operations
DON / Administrator
Keep informed of status and ETA. They manage downtime procedures and staff communication.
T
Response timeline summary
Target response cadence from first report
0–5 min Confirm scope (single device vs. full outage). Check NinjaOne from hotspot. On-site staff / field tech
5–15 min Read hardware lights, check dashboards, identify break point (WAN vs. LAN). Field tech
15 min Notify Anthony. ISP call if WAN is confirmed down. Field tech → AT
20–30 min Activate downtime procedures if clinical systems remain unavailable. DON + field tech
Post-restore Verify all systems online in NinjaOne. Document incident in NorthPoint. AT / field tech