INC-003: Our production server was logging into campus Wi-Fi every 40 seconds

The server reached the internet through the campus captive portal (agnigarh), the same login page students use. A shell script logged it in again and again, because each session lasted about 40...

severity: SEV-1 | kind: outage | date: 2026-03 | duration: Intermittent, through the Student Coordinator → Lead Student Coordinator handover | status: RESOLVED

Impact

CCD's on-campus server had no direct internet. Every time its network session dropped, the apps behind Nginx lost access to everything they depended on, and every CCD portal returned 502 Bad Gateway.

Summary

The server reached the internet through the campus captive portal (agnigarh), the same login page students use. A shell script logged it in again and again, because each session lasted about 40 seconds. Firewall policies kept changing, and whenever the loop broke, everything went down. The real fix wasn't code: emails, repeated visits to the network office and paperwork got the server direct internet access, and then the ports our portals needed.

BEFORE CCD server -> login script (every ~40s) -> campus captive portal -> internet x session drops -> apps lose DB / sign-in / email -> 502 on every portal

AFTER CCD server -> direct access + required outbound opened -> internet

fixed by emails, 5 trips to the network office and paperwork

Timeline

• During the Student Coordinator → Lead Student Coordinator handover, I took over a server that reached the internet only through the campus captive portal
• An auto-login shell script (auto-agnigarh.sh) kept logging the server into the gateway; each session stayed valid for about 40 seconds, so it refreshed constantly
• Campus firewall policies changed several times as the network was tightened; each change could break the login loop
• Whenever the loop failed, the apps couldn't reach their database, sign-in or email services, and every portal behind Nginx showed 502 Bad Gateway
• Wrote to the network admins explaining that a production server can't depend on a 40-second human login
• Followed up in person at the network office 4–5 times and completed the required paperwork
• The server was given direct internet access: no captive portal, no login loop
• Requested the outbound access our portals needed (database, OAuth / sign-in and Microsoft services), which had been blocked; it was opened
• Retired the auto-login script

Root cause

A production server depended on a captive portal designed for people, not machines: 40-second sessions, re-authenticated by a script, on a network whose policies were changing.

What didn't work

• An auto-login loop against the captive portal: it kept things up most of the time, but any hiccup took every portal down
• Adapting the script to each firewall change as it came

Resolution

• Formal request to the network admins, with in-person follow-ups and paperwork
• Direct internet access for the server
• Outbound access opened for database, OAuth / sign-in and Microsoft services
• Login loop retired

Action items

• [x] Server no longer depends on the captive portal
• [ ] Keep a written list of every outbound dependency the portals need, so the next request is one email

Lesson

_Some infrastructure problems aren't technical. The script kept us alive; the emails, the visits to the network office and the paperwork actually fixed it. And a server should never depend on a login meant for a human._

tags: networking, firewall, nginx, linux, campus-infra