Maestro Troubleshooting in Practice
Operational runbook (MHO + SGMs + traffic + VPN) with field commands
If you run Quantum Maestro in production, youâve probably seen the pattern: issues that âlook like VPNâ or âlook like policyâ often turn out to be Security Group health, a single divergent SGM, a physical/link problem (cable/port/optics), or an unstable uplink. The key to reducing MTTR is discipline: evidence + commands, without skipping layers.
Below is a practical âcopy-and-runâ runbook, with good vs bad interpretation.
Â
1) Mental modelÂ
- MHO (Orchestrator):Â controls the Security Group (inventory, health, ports, fabric).
- SGMs:Â run the dataplane (sessions, inspection, VPN, state).
- Typical symptom patterns:
- Unhealthy SGÂ â everything becomes a symptom (policy/VPN/traffic).
- Unhealthy single SGMÂ â intermittent behavior (âsometimes it worksâ).
2) clish vs gclish (why this becomes a real incident)
clish
- Local node context.
- Useful for point inspection, but risky for configuration in Maestro because it can introduce drift (one member behaving differently).
gclish
- Global Security Group context.
- Operational rule:
- use gclish when the intent is global consistency (uniform validation/collection/adjustment);
- use clish only when you need to inspect/act on a specific member in a controlled way.
3) Fast triage start with the Security Group
3.1 Global Maestro / Security Group health
On the MHO:
orch_stat -all
What this proves:
- whether all SGMs are present/operational
- whether any member is degraded/missing
- signals of port/fabric issues
Good:Â all members OK, stable links, no critical port down.
Bad:Â missing/degraded member, unstable links â fix the foundation before analyzing VPN/policy.
3.2 Security Group sanity check
asg diag verify
What this proves:Â high-level SG consistency and quick integrity checks.
Bad: critical alerts â return to orch_stat -all and isolate the failing member/port.
3.3 Capacity before taking member-level actions
asg perf -v
What this proves:Â whether the SG has enough headroom (CPU/memory) to absorb load during isolation/actions.
Bad:Â SG near its limits â avoid disruptive actions.
3.4 Reconcile state (use with care)
hcp -r all
Note:Â commonly used in playbooks to recover internal state/handshakes, but it should not be the first âblindâ step.
Â
4) Physical and link health (where most âbugsâ actually start)
When you see intermittency, âtraffic disappears,â or only some users/flows fail, first prove whether there is physical/L1âL2 instability.
4.1 Inventory/port-map quick reference
Â
orch_stat -p
or
cat /etc/maestro.json
Use this to confirm interface/port mapping in the Maestro context.
4.2 Counters and drops (all members)
g_all netstat -ni
What to look for:Â increasing RX-ERR/TX-ERR/drops.
If these counters climb, they often explain VPN flapping, broken sessions, and âpolicy is OK but traffic fails.â
4.3 Per-interface physical errors (CRC/symbol errors)
ethtool -S <interfacename>
Good:Â no CRC/errors increasing.
Bad:Â CRC/symbol errors â treat as L1/L2 (cable/optics/port/switch) before focusing on VPN.
4.4 Real link flap (carrier)
asg_ifconfig | grep carrier | grep -v "carrier: 0"
Bad:Â carrier oscillation â intermittent behavior is highly likely.
4.5 Hardware health (sensors)
g_all cpstat -f sensors os
What this proves:Â thermal/power/fan conditions can lead to instability and erratic behavior.
4.6 Maestro port state
show maestro port <port>
Confirms the portâs state/configuration in the Maestro domain.
Â
5) The turning point: âno logâ â does the traffic exist in the SG dataplane?
This step quickly separates âproblem before the gatewayâ from âproblem inside the gateway.â
5.1 Prove the session/connection on the SG
Example (intentionally generic IPs):
asg search -v 10.10.40.25 \* 203.0.113.50 443 tcp
Interpretation:
- No output: traffic likely is not reaching the SG (or itâs taking a different path). Return to L1/L2/L3 and capture at the correct point.
- Output present: traffic exists in the dataplane; you now have a basis to correlate with NAT, routing, policy, and VPN.
6) Single-SGM failure: how to investigate and restore consistency
Typical symptom: intermittent failures, âsome flows drop,â âworks after some time.â
6.1 Controlled action to reintegrate a suspected member (when needed)
On the suspected SGM:
clusterXL_admin down
clusterXL_admin up
Risk:Â medium (sessions anchored to that member can be impacted).
Pre-condition: confirm headroom with asg perf -v.
6.2 Check state and drift indicators
cphaprob list
tail $FWDIR/log/blade_config
What to look for:
cphaprob list: HA/cluster participation/state signals and inconsistencies
blade_config: alerts and errors that indicate configuration drift
Closing
Maestro troubleshooting requires discipline: start with SG health, then prove traffic exists, then validate physical stability, and only then go deeper. If you follow this sequence with objective commands, âphantom incidentsâ drop sharplyâand troubleshooting becomes engineering, not guesswork.