Operate & respond
Run NODIF integration predictably.
This public runbook covers the provider side of NODIF. Host access, database credentials, internal queue controls and disaster-recovery commands remain restricted to NODIF platform operators.
Operational ownership
| Layer | Provider owns | NODIF owns |
|---|---|---|
| User/account | Subscription status, limits, payments, HWID/device lifecycle, provider support. | Safe display of compatible status/actions. |
| VPN infrastructure | Gateways, entries, balancers, DNS/TLS for provider services, capacity and abuse handling. | Bounded local route measurement and connection orchestration. |
| Remnawave integration | Rules, External Squad membership, standard metadata, secure panel/API operation. | Issued public slug, branding token, match requirements and integration guidance. |
| NODIF control/analytics plane | Scoped use of portal/API and permitted aggregate decisions. | Availability, TLS, tenant isolation, ingestion, queue processing, databases, retention and backups. |
| Client support | Provider subscription/account problems and provider network incidents. | NODIF app defects, release/signing, tunnel/routing engine and local diagnostics. |
Public service checks
| Check | Expected | Meaning |
|---|---|---|
GET https://panel.nodif.app/health/live | HTTP 200 | API process is running; it does not prove databases are ready. |
GET https://panel.nodif.app/health/ready | HTTP 200 | Required PostgreSQL and ClickHouse dependencies are ready. |
GET https://panel.nodif.app/v1/plans | HTTP 200 JSON | Public API routing/TLS works. |
GET /v1/provider/session | HTTP 200 with valid key | Provider authentication, tenant and plan metadata are available. |
Use a short timeout, low frequency and no credential in URLs. A liveness failure is an incident. A readiness failure can be a dependency incident even if liveness remains 200.
What providers should monitor
- Subscription request success by Remnawave response rule and response type.
- Final presence of exactly one public slug and one branding-token header—measured without storing values.
- Provider gateway/node capacity, connection establishment and upstream availability.
- Canary app import/connect result after Remnawave, proxy/CDN or provider routing changes.
- NODIF provider API status class and safe request ID; never log the Authorization header or response token.
- Aggregate success, P95 latency, loss, jitter and error stages only after a cohort qualifies.
NODIF platform operators separately monitor ingestion rejection, queue pending/ack state, worker failures, database health, backup completion, deletion dispatch and host resources. Providers should escalate symptoms rather than request direct infrastructure access.
Change procedure
- Define scope: provider, environment, External Squad, rule/token/route being changed.
- Record safe baseline: qualified aggregate metrics, health results and canary status; never capture credentials.
- Export rollback material: current Remnawave rules/squad configuration into approved restricted storage.
- Validate staging: syntax, header count/names, fallback and cross-tenant isolation.
- Canary: use dedicated test users and representative network/platform states.
- Observe: wait for subscription refresh and delayed opted-in analytics; watch guardrails.
- Expand in stages: define percentages/segments and a named approver.
- Close: remove temporary access, revoke superseded tokens and retain only redacted evidence.
Incident severity and first actions
| Severity example | First actions | Owner |
|---|---|---|
| Wrong provider branding / suspected cross-tenant header | Stop rollout, remove both NODIF headers from affected scope, revoke exposed token, preserve UTC/request IDs. | Provider incident lead + NODIF security/platform. |
| Provider key or branding token disclosed | Revoke/rotate using another authorized key, remove secret from systems/history, identify scope and time. | Credential owner + security. |
| Widespread subscription import failure | Check Remnawave rule match/fallback, HWID states, provider availability and NODIF readiness; rollback recent provider change. | Provider NOC first, NODIF if app/control-plane implicated. |
| NODIF readiness/API failure | Stop automation retries, record safe request IDs/time and health status, use bounded backoff, escalate. | NODIF platform. |
| Quality regression without import failure | Check provider capacity/routes and qualifying cohorts; pause change; compare baseline and coverage. | Provider network team. |
| Privacy or data-boundary concern | Stop affected processing/export, preserve minimal evidence, notify designated privacy/security contacts. | Joint privacy/security response. |
Safe escalation template
Provider public slug: <public-slug; never tenant UUID unless NODIF explicitly asks>
Environment: staging | production
Impact: <what users/operators cannot do>
Started (UTC): <RFC 3339 timestamp>
Last known good (UTC): <timestamp>
NODIF app version/build: <version>
Platform / OS major / architecture: <coarse values>
Network type: Wi-Fi | mobile | Ethernet | mixed
Stable error code and stage: <coded values>
API request ID(s): <request IDs only>
Recent change: <redacted description>
Canary result and rollback status: <summary>
Reproduction steps: <no subscription/token/host/IP/config>Credential incident runbook
- Classify the credential: provider key, branding token, Remnawave token, subscription token or platform admin key. They are rotated independently.
- Use a different authorized credential to create/verify the replacement.
- Revoke the exposed credential; do not merely delete it from the visible message.
- Search approved logs/repos/tickets by secret scanner or redacted fingerprint—avoid copying plaintext into another tool.
- Check lifecycle metadata such as label, created/last-used/revoked times.
- Assess tenant/scope and time window; do not assume an opaque branding token grants admin access.
- Complete root-cause and preventive action, including history cleanup where supported.
Backup and disaster-recovery boundary
NODIF operates encrypted backups for its production control/analytics data and validates restore material. Providers must independently back up Remnawave, gateway configuration, provider databases, rule exports and secret-manager recovery according to their own RPO/RTO. A NODIF backup does not restore a provider’s Remnawave or VPN nodes.
- Do not send provider database backups or live secrets to NODIF support.
- Test provider restore procedures in an isolated environment.
- After restore, rotate credentials if the backup may contain superseded or exposed values.
- Re-test response-rule ordering, External Squad scoping, HWID and cross-tenant isolation.
Provider rollback checklist
- Canary and rollout assignment stopped.
- Previous known-good response rules/squad configuration restored.
- Both pointer headers removed if integrity/tenant scope is uncertain.
- Disclosed/superseded tokens revoked after safe replacement.
- Generic client fallback and NODIF canary retested.
- Provider gateway health and account/HWID states verified.
- NODIF health and safe API request IDs recorded.
- Post-incident action assigned with no secret evidence retained.