{"id":"2412fba0-cfd2-429c-a6bb-8e86e9febe9f","shortId":"ZtRZLC","kind":"skill","title":"kubesphere-openkruise","tagline":"KubeSphere OpenKruise management Skill. Use when user asks to install or enable OpenKruise, check OpenKruise status, view kruise pods/logs/CRDs, create or update SidecarSet, manage sidecar injection, create or update CloneSet, perform in-place update or batch rollout, uninstall o","description":"# Skill: kubesphere-openkruise\n\n# KubeSphere OpenKruise Management\n\nUse this skill for the full OpenKruise lifecycle in KubeSphere:\n\n- Install or upgrade the OpenKruise extension through `InstallPlan`\n- Query extension status, CRDs, Pods, and logs\n- Generate `SidecarSet` manifests for sidecar injection and rolling updates\n- Generate `CloneSet` manifests for advanced stateless workloads, in-place updates, and batch rollout\n- Uninstall the OpenKruise extension\n- Troubleshoot failed Pods, missing CRDs, and webhook problems\n\nOut of scope by default:\n\n- Advanced OpenKruise CRDs not requested by the user, such as `BroadcastJob`, `NodeImage`, `UnitedDeployment`, or `AdvancedPodAutoscaler`\n- Deep chart-value design without cluster evidence\n\nIf the user explicitly asks for those CRDs, acknowledge that they are OpenKruise capabilities but treat them as a follow-up task instead of assuming they belong in the default workflow.\n\n## Response Rules\n\n- Prefer executable output: `InstallPlan` YAML, `SidecarSet` YAML, `CloneSet` YAML, `kubectl` commands, or a short ordered procedure.\n- Verify the extension name and exact version before generating a final `InstallPlan`.\n- Default the extension name to `openkruise` only if the user context or cluster output does not expose a different resource name.\n- In this KubeSphere environment, the observed mapping is:\n - KubeSphere extension version `1.0.3`\n - OpenKruise runtime version `1.4.0`\n- Distinguish two version concepts before generating `InstallPlan`:\n - KubeSphere extension version: used by `spec.extension.version`\n - OpenKruise runtime version: the controller or component version seen in Pods or docs\n- For this environment, if the user asks to install the current OpenKruise plugin and does not provide another extension version, prefer `spec.extension.version: 1.0.3`.\n- If the user only provides the OpenKruise runtime version, first ask them to confirm the matching KubeSphere extension version.\n- Never invent a version. If the version is missing, first show how to list versions and ask the user to confirm one.\n- `InstallPlan.metadata.name` MUST equal `InstallPlan.spec.extension.name`.\n- `InstallPlan` is cluster-scoped in KubeSphere. Do not add a namespace to `kubectl get|describe|delete installplan`.\n- Use `upgradeStrategy: Manual` unless the user explicitly asks for something else.\n- Omit optional fields instead of guessing values.\n- Before generating `SidecarSet` or `CloneSet`, prefer checking the installed API versions:\n\n```bash\nkubectl api-resources --api-group apps.kruise.io\n```\n\n- If the cluster version is unknown and the user only wants an example, prefer:\n - `SidecarSet`: `apps.kruise.io/v1alpha1`\n - `CloneSet`: `apps.kruise.io/v1alpha1`\n- For uninstall requests, warn that deleting CRDs or CR instances can remove application configuration. Do not suggest deleting CRDs unless the user explicitly asks for full cleanup.\n\n## Version Mapping Discovery\n\nTreat the `1.0.3 -> 1.4.0` mapping as environment evidence, not a universal rule. When the user asks for precision, prove it first:\n\n```bash\n# Discover KubeSphere extension version\nkubectl get extensionversions.kubesphere.io -l kubesphere.io/extension-ref=openkruise\nkubectl get extensionversion openkruise-1.0.3 -o yaml\n\n# Discover deployed runtime image or controller version\nkubectl get pods -n kruise-system -o wide\nkubectl get deploy -n kruise-system kruise-manager -o jsonpath='{.spec.template.spec.containers[*].image}'\nkubectl describe pod -n kruise-system \n```\n\nIf these commands disagree with the assumed mapping, prefer cluster output over the baked-in default.\n\n## Discovery Commands\n\nThis section provides two approaches for querying OpenKruise status:\n1. **KubeSphere API (curl)** - for extension management and multi-cluster queries\n2. **kubectl** - for direct Kubernetes resource operations\n\n### Option 1: Using KubeSphere API (curl)\n\nUse curl with environment variables for querying KubeSphere extension status and multi-cluster resources.\n\n**Environment Variables:**\n```bash\nexport KS_HOST=\"http://\" # KubeSphere console URL (required)\nexport KS_USERNAME=\"admin\" # Username (default: admin)\nexport KS_PASSWORD=\"\" # Password (required)\n```\n\n**Helper Functions (add to ~/.bashrc or use directly):**\n```bash\n# Get OAuth token\nks_token() {\n curl -s -X POST \"$KS_HOST/oauth/token\" \\\n -H \"Content-Type: application/x-www-form-urlencoded\" \\\n -d \"grant_type=password&username=${KS_USERNAME:-admin}&password=$KS_PASSWORD&client_id=kubesphere&client_secret=kubesphere\" | jq -r '.access_token'\n}\n\n# Make API call: ks_api GET/POST/PUT/DELETE [body]\nks_api() {\n local method=${1:-GET}\n local path=$2\n local body=$3\n local token=$(ks_token)\n \n curl -s -X \"$method\" \\\n -H \"Authorization: Bearer $token\" \\\n -H \"Content-Type: application/json\" \\\n ${body:+-d \"$body\"} \\\n \"$KS_HOST$path\"\n}\n```\n\n**Query Commands:**\n\n```bash\n# List all clusters (host + member clusters)\nks_api GET /kapis/cluster.kubesphere.io/v1alpha1/clusters | jq -r '.items[].metadata.name'\n\n# List installed extensions\nks_api GET /kapis/kubesphere.io/v1alpha1/extensions | jq -r '.items[].metadata.name' | grep -i kruise\n\n# List available extension versions\nks_api GET /kapis/kubesphere.io/v1alpha1/extensionversions | jq -r '.items[].metadata.name' | grep -i kruise\n\n# Get OpenKruise extension details\nks_api GET /kapis/kubesphere.io/v1alpha1/extensions/openkruise | jq\n\n# Get cluster connection status (for member clusters)\nks_api GET /kapis/cluster.kubesphere.io/v1alpha1/clusters/member-4/status | jq '.conditions'\n```\n\n### Multi-Cluster Resource Query (host cluster)\n\n```bash\n# Get pods in host cluster\nks_api GET /clusters/host/api/v1/namespaces/kruise-system/pods | jq -r '.items[].metadata.name'\n\n# Get SidecarSets in host cluster (cluster-wide)\nks_api GET /clusters/host/kapis/apps.kruise.io/v1alpha1/users/admin/sidecarsets | jq -r '.items[].metadata.name'\n\n# Get CloneSets in host cluster (cluster-wide)\nks_api GET /clusters/host/kapis/apps.kruise.io/v1alpha1/users/admin/clonesets | jq -r '.items[].metadata.name'\n\n# Get SidecarSets in specific namespace\nks_api GET /clusters/host/kapis/apps.kruise.io/v1alpha1/namespaces/default/sidecarsets | jq -r '.items[].metadata.name'\n\n# Get namespaces where a SidecarSet is applied\nks_api GET /clusters/host/kapis/apps.kruise.io/v1alpha1/users/admin/sidecarsetname/side1 | jq -r '.items[].metadata.name'\n\n# Get specific SidecarSet details\nks_api GET /clusters/host/kapis/apps.kruise.io/v1alpha1/users/admin/sidecarsets/side1 | jq\n\n# Get CRDs in host cluster\nks_api GET /clusters/host/apis/apiextensions.k8s.io/v1/customresourcedefinitions | jq -r '.items[] | select(.metadata.name | contains(\"kruise\")) | .metadata.name'\n```\n\n### Multi-Cluster Resource Query (member cluster)\n\n```bash\n# Get pods in member cluster\nks_api GET /clusters/member-4/api/v1/namespaces/kruise-system/pods | jq -r '.items[].metadata.name'\n\n# Get SidecarSets in member cluster (cluster-wide)\nks_api GET /clusters/member-4/kapis/apps.kruise.io/v1alpha1/users/admin/sidecarsets | jq -r '.items[].metadata.name'\n\n# Get CloneSets in member cluster (cluster-wide)\nks_api GET /clusters/member-4/kapis/apps.kruise.io/v1alpha1/users/admin/clonesets | jq -r '.items[].metadata.name'\n\n# Get CRDs in member cluster\nks_api GET /clusters/member-4/apis/apiextensions.k8s.io/v1/customresourcedefinitions | jq -r '.items[] | select(.metadata.name | contains(\"kruise\")) | .metadata.name'\n```\n\n**API Path Format:**\n\n```\n# For KubeSphere extension management:\n/kapis/kubesphere.io/v1alpha1/extensions\n/kapis/cluster.kubesphere.io/v1alpha1/clusters\n\n# For cluster-wide OpenKruise resources (SidecarSet, CloneSet):\n/clusters/{cluster}/kapis/apps.kruise.io/v1alpha1/users/admin/{resources}\n/clusters/{cluster}/kapis/apps.kruise.io/v1alpha1/users/admin/{resources}/{name}\n/clusters/{cluster}/kapis/apps.kruise.io/v1alpha1/users/admin/sidecarsetname/{sidecarsetName}\n\n/clusters/{cluster}/api/v1/namespaces/{namespace}/{resources}\n/clusters/{cluster}/apis/apiextensions.k8s.io/v1/customresourcedefinitions\n```\n\n**Query Parameters:**\n- `page` - Page number (default: 1)\n- `limit` - Items per page\n- `ascending` - Sort direction (default: false)\n- `sortBy` - Sort field (e.g., createTime)\n\n### Option 2: Using kubectl (Direct Cluster Access)\n\nUse `kubectl` for direct Kubernetes resource operations on the **host cluster**.\n\n```bash\n# Extension and version discovery in KubeSphere\nkubectl get extensions.kubesphere.io | grep -i kruise\nkubectl get extensionversions.kubesphere.io | grep -i kruise\nkubectl get extensionversions.kubesphere.io -l kubesphere.io/extension-ref=openkruise\n\n# InstallPlan and extension status\nkubectl get installplans.kubesphere.io\nkubectl get installplan openkruise -o yaml\nkubectl describe extension openkruise\nkubectl describe extensionversion openkruise-\n\n# OpenKruise runtime resources\nkubectl api-resources --api-group apps.kruise.io\nkubectl get crd | grep -E 'clonesets.apps.kruise.io|sidecarsets.apps.kruise.io'\nkubectl get pods -A | grep -i kruise\nkubectl get sidecarsets.apps.kruise.io -A\nkubectl get clonesets.apps.kruise.io -A\n```\n\n### Option 3: Multi-Cluster Query (kubeconfig extraction)\n\nFor querying **member clusters**, extract the kubeconfig from the Cluster resource:\n\n```bash\n# Get kubeconfig for a member cluster\nCLUSTER_NAME=member-4\nKUBECONFIG_ENCODED=$(kubectl get cluster.cluster.kubesphere.io $CLUSTER_NAME -o jsonpath='{.spec.connection.kubeconfig}')\necho \"$KUBECONFIG_ENCODED\" | base64 -d > /tmp/${CLUSTER_NAME}-kubeconfig\n\n# Query member cluster\nexport KUBECONFIG=/tmp/${CLUSTER_NAME}-kubeconfig\nkubectl get pods -n kruise-system\nkubectl get sidecarsets.apps.kruise.io -A\nkubectl get clonesets.apps.kruise.io -A\n\n# Switch back to host cluster\nexport KUBECONFIG=\"\"\n```\n\n### When to Use Which Approach\n\n| Scenario | Recommended Approach |\n|----------|---------------------|\n| Query KubeSphere extension status | KubeSphere API (curl) |\n| List available clusters | KubeSphere API (curl) |\n| Query host cluster Kubernetes resources | kubectl |\n| Query member cluster Kubernetes resources | kubeconfig extraction |\n| Create/apply InstallPlan/SidecarSet/CloneSet | kubectl |\n| Get extension version info | KubeSphere API (curl) |\n\n## Install OpenKruise in KubeSphere\n\n### 1. Pre-check\n\n```bash\nkubectl get extension openkruise\nkubectl get extensionversions.kubesphere.io -l kubesphere.io/extension-ref=openkruise\nkubectl describe extensionversion openkruise-\n```\n\nCheck:\n\n- The extension resource really exists\n- The exact extension version exists\n- Whether the extension is single-cluster or multi-cluster\n- Whether the user needs custom config\n- Whether the user is giving an extension version or only the OpenKruise runtime version\n- In this environment, remember that extension `1.0.3` maps to OpenKruise runtime `1.4.0`\n\n### 2. InstallPlan Template\n\nUse this when the user has already confirmed the extension name and version.\n\n**Note:** \n- InstallPlan is a KubeSphere CRD resource, **created and managed only in the host cluster** (not in member clusters)\n- `clusterScheduling.placement.clusters` specifies which clusters the extension **agent components** should be installed to (host, member-4, etc.)\n\n```yaml\napiVersion: kubesphere.io/v1alpha1\nkind: InstallPlan\nmetadata:\n name: openkruise\nspec:\n enabled: true\n extension:\n name: openkruise\n version: \n upgradeStrategy: Manual\n # config: |\n # featureGates: \"PreDownloadImageForInPlaceUpdate=true\"\n clusterScheduling:\n placement:\n clusters:\n - host\n - member-4\n # overrides:\n # host: |-\n # featureGates: \"PreDownloadImageForInPlaceUpdate=true\"\n```\n\n**Fields:**\n- `clusterScheduling.placement.clusters`: List of clusters to install the extension **agent** (e.g., `host`, `member-4`). The OpenKruise agent will be deployed to these clusters.\n- `clusterScheduling.overrides`: Cluster-specific config overrides (optional)\n\nApply and verify:\n\n```bash\nkubectl apply -f installplan-openkruise.yaml\nkubectl get installplan openkruise -w\nkubectl describe installplan openkruise\nkubectl get extension openkruise -o yaml\n```\n\n**Check installation status per cluster:**\n\n```bash\nkubectl get installplan openkruise -o jsonpath='{.status.clusterSchedulingStatuses}'\n```\n\n### 2.1 Config Handling\n\n`spec.config` is a YAML string block, not a nested object. Only include it when the user asks for non-default settings.\n\n```yaml\nspec:\n config: |\n featureGates: \"PreDownloadImageForInPlaceUpdate=true\"\n```\n\nRules:\n\n- The content under `config: |` must itself be valid YAML\n- Do not invent config keys without evidence from chart values, extension documentation, or user-provided requirements\n- If config keys are unknown, prefer default install and show discovery commands first\n\nUseful discovery commands:\n\n```bash\nkubectl describe extensionversion openkruise-\nkubectl get extensionversion openkruise- -o yaml\n```\n\n### 3. Status and Logs After Install\n\n```bash\nTARGET_NS=$(kubectl get installplan openkruise -o jsonpath='{.status.targetNamespace}')\nJOB_NAME=$(kubectl get installplan openkruise -o jsonpath='{.status.jobName}')\n\nkubectl get pods -n \"${TARGET_NS:-kruise-system}\"\nkubectl logs -n \"${TARGET_NS:-kruise-system}\" -l job-name=\"$JOB_NAME\" --tail=200\nkubectl logs -n \"${TARGET_NS:-kruise-system}\" deploy/kruise-manager --tail=200\nkubectl get crd | grep -E 'clonesets.apps.kruise.io|sidecarsets.apps.kruise.io'\n```\n\n## Manage SidecarSet\n\nUse `SidecarSet` when the user wants cluster-level sidecar injection and separate lifecycle management for sidecars.\n\n### SidecarSet Template\n\n```yaml\napiVersion: apps.kruise.io/v1alpha1\nkind: SidecarSet\nmetadata:\n name: log-agent\nspec:\n selector:\n matchLabels:\n app: demo\n namespaceSelector:\n matchLabels:\n kubernetes.io/metadata.name: demo-project\n containers:\n - name: log-agent\n image: fluent/fluent-bit:2.2\n imagePullPolicy: IfNotPresent\n env:\n - name: LOG_LEVEL\n value: info\n volumeMounts:\n - name: varlog\n mountPath: /var/log\n podInjectPolicy: BeforeAppContainer\n volumes:\n - name: varlog\n hostPath:\n path: /var/log\n updateStrategy:\n type: RollingUpdate\n maxUnavailable: 1\n```\n\nCommon operations:\n\n```bash\nkubectl apply -f sidecarset.yaml\nkubectl get sidecarset log-agent -o yaml\nkubectl describe sidecarset log-agent\nkubectl delete sidecarset log-agent\n```\n\nVerify injection:\n\n```bash\nkubectl get pods -n demo-project -l app=demo\nkubectl get pod -n demo-project -o jsonpath='{.spec.containers[*].name}'\nkubectl describe pod -n demo-project\n```\n\nSidecarSet troubleshooting checks:\n\n- `spec.selector` matches the workload Pod labels\n- `namespaceSelector` matches the target namespace labels\n- The sidecar image can be pulled in target namespaces\n- `kruise-manager` webhook is healthy\n\n## Manage CloneSet\n\nUse `CloneSet` when the user wants an enhanced workload with in-place image updates, batch rollout, or richer update controls than `Deployment`.\n\n### CloneSet Template\n\n```yaml\napiVersion: apps.kruise.io/v1alpha1\nkind: CloneSet\nmetadata:\n name: sample-app\n namespace: demo-project\nspec:\n replicas: 3\n selector:\n matchLabels:\n app: sample-app\n template:\n metadata:\n labels:\n app: sample-app\n spec:\n containers:\n - name: app\n image: nginx:1.25\n imagePullPolicy: IfNotPresent\n ports:\n - containerPort: 80\n updateStrategy:\n type: InPlaceIfPossible\n partition: 0%\n maxUnavailable: 1\n```\n\nCommon operations:\n\n```bash\nkubectl apply -f cloneset.yaml\nkubectl get cloneset sample-app -n demo-project\nkubectl describe cloneset sample-app -n demo-project\nkubectl delete cloneset sample-app -n demo-project\n```\n\nVerify rollout state:\n\n```bash\nkubectl get pods -n demo-project -l app=sample-app -o wide\nkubectl get events -n demo-project --sort-by=.lastTimestamp\n```\n\n### In-place Update Guidance\n\nWhen the user asks for \"in-place update\" or an in-place image update:\n\n1. Keep `spec.updateStrategy.type: InPlaceIfPossible`\n2. Change the container image in `spec.template.spec.containers`\n3. Re-apply the manifest\n\n```bash\nkubectl get cloneset sample-app -n demo-project -o yaml > cloneset.yaml\n# edit image and, if needed, updateStrategy.partition\nkubectl apply -f cloneset.yaml\nkubectl describe cloneset sample-app -n demo-project\nkubectl get pods -n demo-project -l app=sample-app -o wide\n```\n\n### Batch Rollout Guidance\n\nUse `partition` to keep part of the Pods on the old revision during a rollout.\n\n- `partition: 100%` means keep all replicas on the old revision\n- `partition: 50%` means only half of the replicas move to the new revision\n- `partition: 0%` means finish the rollout\n\nExample rollout sequence:\n\n1. Change image to the new version and set `partition: 100%`\n2. Lower to `partition: 50%`\n3. Lower to `partition: 0%`\n\nAt each step, verify Pod revision movement:\n\n```bash\nkubectl get cloneset sample-app -n demo-project -o yaml\nkubectl get pods -n demo-project -l app=sample-app -o wide\n```\n\n### CloneSet Troubleshooting Checks\n\n- If Pods are recreated instead of in-place updated, confirm `updateStrategy.type` is `InPlaceIfPossible` or `InPlaceOnly`\n- Some field changes still require Pod recreation\n- If CPU and memory are changed together with image updates in unsupported environments, split the change into separate steps\n- Check the controller and workload events with:\n\n```bash\nkubectl describe cloneset sample-app -n demo-project\nkubectl get pods -n demo-project -l app=sample-app -o wide\nkubectl get events -n demo-project --sort-by=.lastTimestamp\n```\n\n## Uninstall OpenKruise\n\n### Uninstall Pre-check\n\nCheck whether workloads still depend on OpenKruise resources before uninstalling the extension:\n\n```bash\nkubectl get sidecarsets.apps.kruise.io -A\nkubectl get clonesets.apps.kruise.io -A\n```\n\nIf these resources still exist, tell the user to migrate or remove them first.\n\n### Default uninstall path:\n\n```bash\nkubectl delete installplan openkruise\nkubectl get installplan openkruise\nkubectl get pods -n kruise-system | grep -i kruise\n```\n\nIf the user wants full cleanup, remind them to handle application resources first:\n\n- Delete or migrate `SidecarSet` and `CloneSet` resources that are still in use\n- Confirm whether application teams still depend on OpenKruise behavior\n- Only discuss CRD removal after the user explicitly confirms full cleanup\n\n## Troubleshooting Playbook\n\n### 1. InstallPlan or extension failed\n\n```bash\nkubectl describe installplan openkruise\nkubectl get installplan openkruise -o jsonpath='{.status.conditions}'\nkubectl get extension openkruise -o yaml\nkubectl get extensionversion openkruise- -o yaml\n```\n\n### 2. Kruise Pods are unhealthy\n\n```bash\nkubectl get pods -A | grep -i kruise\nkubectl describe pod -n kruise-system \nkubectl logs -n kruise-system deploy/kruise-manager --tail=200\nkubectl get events -n kruise-system --sort-by=.lastTimestamp\n```\n\n### 3. CRD missing or not established\n\n```bash\nkubectl get crd clonesets.apps.kruise.io sidecarsets.apps.kruise.io\nkubectl describe crd clonesets.apps.kruise.io\nkubectl describe crd sidecarsets.apps.kruise.io\nkubectl api-resources --api-group apps.kruise.io\n```\n\nLikely causes:\n\n- OpenKruise extension installation did not finish\n- CRDs were partially created or rejected by the API server\n- Webhook or controller startup failed before CRDs became usable\n\nSafe next actions:\n\n- Re-check `InstallPlan` state and conditions\n- Inspect install Job logs and `kruise-manager` logs\n- Avoid creating workload CRs until the CRDs are established\n\n### 4. Webhook failure\n\n```bash\nkubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations | grep -i kruise\nkubectl describe mutatingwebhookconfiguration | grep -i -A5 kruise\nkubectl describe validatingwebhookconfiguration | grep -i -A5 kruise\nkubectl logs -n kruise-system deploy/kruise-manager --tail=200\n```\n\nLikely causes:\n\n- `kruise-manager` Pod is not Ready\n- Webhook service endpoints are missing\n- TLS certificate has expired or does not match the service DNS name\n- The API server cannot reach the webhook service\n- The target Pod or namespace does not match `SidecarSet` selectors\n\nSafe next actions:\n\n- Check `kruise-manager` Pod readiness and recent restarts\n- Check Service and Endpoints for the webhook target\n- Inspect recent TLS or x509 errors in controller logs\n- Verify the target namespace labels and Pod labels match the `SidecarSet`\n\n### 5. Sidecar not injected\n\n```bash\nkubectl get sidecarset -A\nkubectl get sidecarset -o yaml\nkubectl get pod -n -o yaml\nkubectl get namespace --show-labels\n```\n\n### 6. CloneSet rollout stuck\n\n```bash\nkubectl describe cloneset -n \nkubectl get pods -n -l app=