Skip to content

Runbook: Keycloak Realm Recovery ​

Restore a broken or misconfigured Keycloak realm on the Microtec ERP platform.

Audience: Platform engineers, DevOps
Prerequisites: Keycloak admin credentials, az CLI authenticated
Container: mic-erp-be-{env}-keycloak in the Public CAE


Overview ​

Microtec ERP uses Keycloak with custom SPIs (Service Provider Interfaces) called keycloak-seeding. These SPIs run automatically on every Keycloak startup and are responsible for:

  • Creating realms (microtec for ERP, businessowner for BO portal)
  • Configuring OIDC clients per service
  • Setting authentication flow priorities
  • Enabling/disabling specific clients

Because seeding runs on startup, most realm recovery scenarios are solved by restarting the Keycloak container.


Realm Structure ​

RealmAudienceKey Clients
microtecERP users (all modules)apps-portal, inventory, hr, accounting, gateway
businessownerTenant admin portalbusiness-owners-app, business-owners-api

Scenario 1: Realm is Missing or Corrupted ​

Symptoms ​

  • Login page returns "Realm not found"
  • Keycloak admin shows the realm is absent or incomplete
  • Services return 401 with "Invalid realm" in logs

Resolution — Restart Keycloak (seeding auto-recovers) ​

bash
export ENV="dev"   # or stage, preprod, uat, prod
export RG="mic-erp-be-${ENV}-containers-rg"
export KC_APP="mic-erp-be-${ENV}-keycloak"

# [ACTION] Get the active revision
REVISION=$(az containerapp revision list \
  --name "${KC_APP}" --resource-group "${RG}" \
  --query "[?properties.active==\`true\`].name" -o tsv)

# [ACTION] Restart — seeding SPI runs on startup and recreates the realm
az containerapp revision restart \
  --name "${KC_APP}" \
  --resource-group "${RG}" \
  --revision "${REVISION}"

echo "Keycloak restarting... allow 60-90 seconds for full startup + seeding"
bash
# [VERIFY] Confirm realm exists after restart
sleep 90
curl -sf "https://{domain}/auth/realms/microtec" | jq .realm
# Expected: "microtec"

curl -sf "https://{domain}/auth/realms/businessowner" | jq .realm
# Expected: "businessowner"

Scenario 2: Authentication Flow Priorities Broken ​

Symptoms ​

  • Users can log in with password but MFA or IdP-linked login fails
  • Cookie-based session not honoured
  • MAC (Microtec Auth Cookie) flow not triggered

Background ​

Keycloak authentication flows use a priority order. The Microtec platform requires:

FlowPriorityPurpose
MAC (Microtec Auth Cookie)10Honour existing Microtec session cookie
Cookie20Standard Keycloak cookie-based SSO
IdP Redirect30Identity Provider (external) login
Forms40+Username/password form

[WARNING] If priorities are wrong (e.g., IdP=10 when it should be 30), IdP login will intercept all requests and password login may become unreachable.

Fix — Restart triggers priority restoration ​

The seeding SPI's ConfigureRealmClients step restores priorities on startup. Restart Keycloak as described in Scenario 1.

Manual fix via Keycloak Admin API (if restart is not possible) ​

bash
export KC_URL="https://{domain}/auth"
export REALM="microtec"

# [ACTION] Get admin token
ADMIN_TOKEN=$(curl -s -X POST \
  "${KC_URL}/realms/master/protocol/openid-connect/token" \
  -d "client_id=admin-cli" \
  -d "username=admin" \
  -d "password=${KC_ADMIN_PASS}" \
  -d "grant_type=password" | jq -r .access_token)

# [ACTION] List authentication flows
curl -s -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  "${KC_URL}/admin/realms/${REALM}/authentication/flows" | jq '.[] | {alias, id}'

# [ACTION] Update MAC flow execution priority to 10
# (Replace FLOW_ID and EXECUTION_ID with values from the list above)
curl -s -X PUT \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  "${KC_URL}/admin/realms/${REALM}/authentication/flows/${FLOW_ID}/executions" \
  -d '{"id":"EXECUTION_ID","priority":10}'

Scenario 3: Client Disabled or Missing ​

Symptoms ​

  • Specific service returns 401 with "Client not found" or "Client disabled"
  • Keycloak admin shows the client as disabled

Background ​

The seeding SPI's ConfigureRealmClients step:

  • Disables account-console (not needed in ERP)
  • Ensures account client is enabled (required for token introspection)
  • Enables all ERP service clients

SetClientEnabledAsync is called per client during seeding. Again, restart recovers this automatically.

Manual fix via Admin API ​

bash
# [ACTION] Get client ID (internal UUID)
CLIENT_UUID=$(curl -s -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  "${KC_URL}/admin/realms/${REALM}/clients?clientId=apps-portal" | jq -r '.[0].id')

# [ACTION] Enable the client
curl -s -X PUT \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  "${KC_URL}/admin/realms/${REALM}/clients/${CLIENT_UUID}" \
  -d "{\"enabled\": true}"

echo "Client apps-portal re-enabled"

Scenario 4: Backchannel Logout Broken ​

Symptoms ​

  • User logout from one app does not propagate to other apps (SSO session persists)
  • Logs show "Backchannel logout URL unreachable"

Background ​

Backchannel logout is already configured in the seeding SPI (ConfigureBackchannelLogout). The logout URL for each service is set to {service-internal-url}/auth/logout.

Verify the internal FQDN is reachable from the Keycloak container:

bash
# [ACTION] Execute inside Keycloak container
az containerapp exec \
  --name "${KC_APP}" \
  --resource-group "${RG}" \
  --command "curl -s http://mic-erp-be-dev-apps-portal/health"
# Expected: {"status":"Healthy"}

If unreachable, the Private CAE mTLS or DNS resolution is misconfigured — escalate to the platform team.


Scenario 5: Full Realm Data Loss ​

[WARNING] This is a destructive recovery. All realm customisations not covered by the seeding SPI will be lost.

bash
# [ACTION] Delete the broken realm
curl -s -X DELETE \
  -H "Authorization: Bearer ${ADMIN_TOKEN}" \
  "${KC_URL}/admin/realms/${REALM}"

# [ACTION] Restart Keycloak — seeding recreates the realm from scratch
az containerapp revision restart \
  --name "${KC_APP}" --resource-group "${RG}" \
  --revision "${REVISION}"

After restart, verify both realms are recreated and all clients are present.


Keycloak Admin Console Access ​

EnvironmentAdmin Console URL
devhttps://dev.microtec-test.com/auth/admin
stagehttps://stage.microtecstage.com/auth/admin
prodhttps://onlinemicrotec.com.sa/auth/admin

[INFO] Admin credentials are stored in Key Vault under Keycloak--AdminPassword.

bash
# [ACTION] Retrieve admin password from KV
az keyvault secret show \
  --vault-name "mic-erp-be-dev-skv" \
  --name "Keycloak--AdminPassword" \
  --query "value" -o tsv

Startup Sequence Reference ​

When Keycloak starts, the seeding SPI executes in this order:

1. CreateRealm              — Create realm if not exists
2. ConfigureRealmSettings   — Token lifetime, brute force protection
3. CreateClients            — Register all OIDC clients
4. SetClientEnabledAsync    — Enable/disable specific clients
5. ConfigureRealmClients    — Set flow priorities, disable account-console
6. ConfigureAuthFlows       — Create/update MAC / Cookie / IdP flows
7. ConfigureBackchannelLogout — Set logout URLs
8. CreateDefaultUsers       — Seed test users (non-prod only)

Monitoring Keycloak ​

bash
# [INFO] Follow Keycloak startup logs
az containerapp logs show \
  --name "${KC_APP}" --resource-group "${RG}" \
  --follow --tail 200

# Look for these log lines to confirm successful seeding:
# INFO  [...keycloak-seeding] Realm 'microtec' seeding completed
# INFO  [...keycloak-seeding] Realm 'businessowner' seeding completed

Internal Documentation — Microtec