Agentic AMS Platform

AMS Architecture

Overview

The Agentic AMS Framework is a multi-agent system. It detects production anomalies. It retrieves relevant remediation procedures from an organizational knowledge base. It executes a remediation plan through real infrastructure tooling. It validates the result. A human approval gate is between planning and execution. No irreversible action happens without operator consent.

Problem

Grafana alerts wake on-call engineers at 3 AM. The engineers then spend significant time tracing causation through logs across distributed services. The alert indicates that something is wrong. It does not explain what is wrong, why it is wrong, or how to fix it. The system automates the detection-to-resolution pipeline. It keeps a human in the loop for the decisions that matter.

Approach

The system is organized into four layers. Each layer has a distinct responsibility.

Observability. Application logs flow into Grafana Loki. Grafana Loki triggers keyword-based alerting rules. An Observability Agent connects to the dashboard over a persistent WebSocket. It reads alerts as they arrive. It checks ServiceNow for existing tickets. If no ticket exists, it creates one. If a ticket already exists, it skips duplicate creation. This deduplication logic prevents downstream agents from processing the same alert repeatedly during flapping events.

Knowledge retrieval. An agent that detects problems but cannot fix them is only a notifier. The RAG pipeline uses Qdrant. Qdrant is a vector database. It holds the organization's Standard Operating Procedures sourced from Confluence via batch ingestion. When the Agent Orchestrator receives an incident, it fetches the most semantically relevant SOP. The agents then reason with institutional knowledge rather than from scratch.

Planning and execution. Three agents handle remediation:

• The Planning Agent receives the incident description and the SOP that was retrieved. It produces an ordered remediation plan. It does not execute anything.
• A Human-in-the-Loop checkpoint then presents the proposed plan to a human operator. The operator approves or rejects the plan. This checkpoint is not ceremonial. It is the mechanism that earns the system the right to be trusted with real infrastructure.
• Upon approval, the Executor Agent works through the plan step by step. It calls tools. The tools are HttpRequest for API interactions, JenkinsRollback for deployment reversions, and ReadPodStatus for Kubernetes pod queries. After each step, the Validator Agent independently verifies whether the step had its intended effect. If validation fails, the Executor retries. If it succeeds, the pipeline advances.

Resolution. When all steps pass validation, the ServiceNow ticket is closed. The system then notifies the on-call engineer.

How It Works

1. Grafana alerts trigger on anomalous log patterns.
2. The Observability Agent reads the alert. It checks ServiceNow for an existing ticket. If needed, it creates one.
3. The Agent Orchestrator fetches the most relevant SOP from Qdrant.
4. The Planning Agent produces an ordered remediation plan based on the incident and the SOP.
5. A human operator reviews and approves (or rejects) the plan.
6. The Executor Agent runs each plan step using infrastructure tools.
7. The Validator Agent verifies each step's outcome.
8. On full resolution, the system closes the ServiceNow ticket.

Tech Stack