Overview
The Agentic AMS Framework is a multi-agent system for production incidents. It detects anomalies, retrieves remediation procedures from the knowledge base, and runs remediation through infrastructure tools. It validates each result, with a human approval gate between planning and execution. No irreversible action runs without operator approval.
Problem
Grafana alerts wake on-call engineers at 3 AM, forcing them to trace causation through logs across distributed services. The alert shows that something is wrong, but not what is wrong, why it is wrong, or how to fix it. The system automates the detection-to-resolution pipeline, keeping a human in the loop for decisions that matter.
Approach
The system has four layers, each with a distinct responsibility.
Observability
Application logs flow into Grafana Loki, triggering keyword-based alert rules. An Observability Agent connects to the dashboard over a persistent WebSocket, reading alerts as they arrive. It checks ServiceNow for existing tickets, creating one only when no ticket exists. This check stops downstream agents from processing the same alert repeatedly during flapping events.
Knowledge retrieval
An agent that detects problems but cannot fix them is only a notifier. The RAG pipeline uses Qdrant, a vector database holding Standard Operating Procedures from Confluence. Batch ingestion loads the procedures in advance. When the Agent Orchestrator receives an incident, it fetches the most relevant SOP, letting agents reason with institutional knowledge.
Planning and execution
Three agents handle remediation:
- The Planning Agent receives the incident description and the retrieved SOP, producing an ordered remediation plan without running any action.
- A Human-in-the-Loop checkpoint then shows the proposed plan to a human operator for approval or rejection. This checkpoint is not ceremonial, giving the system approval to use real infrastructure.
- Upon approval, the Executor Agent runs the plan step by step, calling HttpRequest for APIs, JenkinsRollback for reversions, and ReadPodStatus for pod queries. After each step, the Validator Agent verifies the effect, retrying on failure and continuing on success.
Resolution
When all steps pass validation, the system closes the ServiceNow ticket and notifies the on-call engineer.
How it works
- Grafana alerts trigger on anomalous log patterns.
- The Observability Agent reads the alert and checks ServiceNow, creating a ticket only when needed.
- The Agent Orchestrator fetches the most relevant SOP from Qdrant.
- The Planning Agent produces an ordered remediation plan from the incident and the SOP.
- A human operator reviews the plan and approves or rejects it.
- The Executor Agent runs each plan step with infrastructure tools.
- The Validator Agent verifies each step outcome, triggering retries on failure.
- On full resolution, the system closes the ServiceNow ticket.
Tech stack
- Qdrant
- Grafana
- FastAPI
- Python