Building for a live ISP network
For the last decade, I have built software for ISP operators, including FCC compliance platforms, broadband label tooling, fault automation and network provisioning. I have seen those tools meet the conditions of a live NOC during an overnight incident.
AI appliance announcements put capable models within reach of network operators. The deployment still needs to handle the equipment, errors and operating conditions of a live network.
I believe in the category. I'm building in it. But I've watched what happens when systems that work in a controlled demo meet actual operational environments, and the failure modes of autonomous agents in network operations are more consequential than most system failures I've debugged.
A production appliance has to handle a Nokia OLT fault, recover from transient errors, avoid repeating a completed configuration change and preserve an audit trail.
THE GAP BETWEEN "IT RUNS" AND "IT'S RELIABLE"
Production exposes conditions that staging may miss: changing load, intermittent connections, unexpected device state and concurrent human activity. Recovery has to account for what actually happened before an operation is retried.
Operational software already needs to handle service crashes, database timeouts and network partitions with retries, circuit breakers and health checks. Agentic workflows also require checks on the meaning and consequences of a proposed action.
AI agents fail differently. An agent can produce a subtly wrong answer with high confidence. It can execute the right command in the wrong order. It can interpret ambiguous state information differently than a human would. These are behavioral failures rather than crashes, and they look like success until the consequences arrive.
For an ISP running automated fault resolution, a behavioral failure does more than produce a wrong log entry. It executes a command on live equipment, and the consequence can be a service outage for hundreds of subscribers rather than a simple service restart.
I spent years at ETI watching this problem play out in real networks. I've listened to operators describe incidents where a half-correct automation decision cascaded into a broader outage. That's what I'm designing against in XSI LodeStone™.
WHAT PRODUCTION RELIABILITY ACTUALLY REQUIRES
Four requirements shape that recovery and execution design:
- Idempotency and state management. When an action stops partway through, the system needs to establish what completed before it retries, compensates or escalates. A configuration change may not be fully reversible. The application must retain enough action state to recover without repeating a completed operation.
- Circuit breakers and backpressure. Production Kubernetes environments impose rate limits at multiple layers. The agent framework must implement circuit breakers that detect when downstream systems are degraded, back off gracefully, and queue actions rather than overwhelming a management interface that's already stressed. Without this, an agent responding to a network incident can amplify the incident.
- Audit trails and rollback. Every action an agent takes needs a complete, append-only audit record: what query was submitted, what model was consulted, what response was received, what action was taken, what the result was. For regulated operators, such as ISPs with CPNI obligations or operators in BEAD compliance cycles, this record is what lets them demonstrate what the system did and why. The record also supports investigation of behavioral failures after the fact.
- Graduated autonomy with conservative defaults. Read operations need permission for the relevant data. Write operations need a defined scope, validation and a record of the result. Service-affecting changes need the applicable operator authorization before execution.
Before you retry
Select diagram to enlargeTHE K3S ARCHITECTURE DECISION
I selected K3s for XSI LodeStone’s infrastructure to fit Kubernetes workloads into an edge deployment.
Full K8s on a single-node appliance is operationally expensive. The control plane overhead (API server, etcd, controller manager, scheduler) consumes significant memory on a device where every gigabyte is competing with model inference. The operational complexity of running a full cluster on embedded hardware creates fragility that contradicts the "plug it in and it works" product promise.
K3s is a lightweight Kubernetes distribution designed for environments such as edge and resource-constrained deployments. Resource use depends on the configuration and workload. Migration to another Kubernetes distribution still requires checks for storage, networking, versions and deployment-specific behavior.
The deployment needs to recover from power cycles and software updates and expose failures an operator can diagnose. Control-plane maintenance competes with inference for the appliance’s resources.
| REQUIREMENT | DEMO ENVIRONMENT | PRODUCTION ISP NETWORK |
|---|---|---|
| Uptime | Hours | Defined service objective |
| Failure Recovery | Manual intervention | Tested recovery and escalation |
| Audit Trails | Optional logging | Append-only, supports compliance obligations |
| Behavioral Failure Handling | Re-run the agent | Idempotent, state-aware recovery |
| Configuration Management | Manual or ad-hoc | Reviewed desired configuration |
| Load Handling | Single user, controlled | Circuit breakers, graceful degradation |
GITOPS: WHY CONFIGURATION DRIFT IS A PRODUCTION KILLER
Configuration drift can make production differ from the system that was tested. A manual flag change or resource adjustment can leave the actual deployment out of step with its intended configuration.
XSI LodeStone’s deployment design uses GitOps and Argo CD to manage reviewed configuration. Differences between desired and actual state can then be detected and handled through the deployment policy.
Across a fleet of customer sites, a recorded configuration and repeatable update process help the support team investigate differences without reconstructing every manual change.
WHAT SEPARATES THE SURVIVORS
The deployment has to support reliable operation, investigation and recovery across the work it is authorized to perform.
A useful model result is one part of that job. The application also needs to manage state, recover safely, preserve evidence and give operators control over consequential changes.
Rhyan J. Neble | Founder & CEO, Extended Systems Intelligence | rneble@xtendedsystems.com | xsilodestone.ai
