Kubeeye – Tool To Find Various Problems On Kubernetes, Such As Application Misconfiguration, Unhealthy Cluster Components And Node Problems

[]()

KubeEye is an inspection tool for [Kubernetes]( “Kubernetes” ) to discover Kubernetes resources (by [OPA]( “OPA” ) ), cluster components, cluster nodes (by [Node-Problem-Detector]( “Node-Problem-Detector” )) and other configurations are meeting with best practices, and giving suggestions for modification.

KubeEye supports custom inspection rules and plugins installation. Through [KubeEye Operator]( “KubeEye Operator” ), you can view the inspection results and [modification]( “modification” ) suggestions by the graphical display on the web page.

## Architecture

KubeEye get cluster resource details by the Kubernetes API, inspect the resource configurations by inspection rules and plugins, and generate inspection results. See Architecture for details.

[]()

## How to use

* Install KubeEye on your machine

* Download pre built executables from [Releases]( “Releases” ).

* Or you can build from source code

> Note: make install will create kubeeye in /usr/local/bin/ on your machine.

git clone https://github.com/kubesphere/kubeeye.git
cd kubeeye
make installke

* [Optional] Install [Node-problem-Detector]( “Node-problem-Detector” )

> Note: This will install npd on your cluster, only required if you want detailed report.

kubeeye install npd

* Run KubeEye

> Note: The results of kubeeye sort by resource kind.

kubeeye audit
KIND NAMESPACE NAME REASON LEVEL MESSAGE
Node docker-desktop kubelet has no sufficient memory available warning KubeletHasNoSufficientMemory
Node docker-desktop kubelet has no sufficient PID available warning KubeletHasNoSufficientPID
Node docker-desktop kubelet has disk pressure warning KubeletHasDiskPressure
Deployment default testkubeeye NoCPULimits
Deployment default testkubeeye NoReadinessProbe
Deployment default testkubeeye NotRunAsNonRoot
Deployment kube-system coredns NoCPULimits
Deployment kube-system coredns ImagePullPolicyNotAlways
Deployment kube-system coredns NotRunAsNonRoot
Deployment kubeeye-system kubeeye-controller-manager ImagePullPolicyNotAlways
Deployment kubeeye-system kubeeye-controller-manager NotRunAsNonRoot
DaemonSet kube-system kube-proxy NoCPULimits
DaemonSet k ube-system kube-proxy NotRunAsNonRoot
Event kube-system coredns-558bd4d5db-c26j8.16d5fa3ddf56675f Unhealthy warning Readiness probe failed: Get “https://10.1.0.87:8181/ready”: dial tcp 10.1.0.87:8181: connect: connection refused
Event kube-system coredns-558bd4d5db-c26j8.16d5fa3fbdc834c9 Unhealthy warning Readiness probe failed: HTTP probe failed with statuscode: 503
Event kube-system vpnkit-controller.16d5ac2b2b4fa1eb BackOff warning Back-off restarting failed container
Event kube-system vpnkit-controller.16d5fa44d0502641 BackOff warning Back-off restarting failed container
Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f5fc3229c Failed warning Failed to pull image “controller:latest”: rpc error: code = Unknown desc = Error response from daemon: pull access denied for controller, repository does not exist or may require ‘docker login’: denied: requested access to the resource is denied
Event kubeeye-system kubeeye-controller-manager-7f79c4ccc8-f2njw.16d5fa3f61b28527 Failed warning Error: ImagePullBackOff
Role kubeeye-system kubeeye-leader-election-role CanDeleteResources
ClusterRole kubeeye-manager-role CanDeleteResources
ClusterRole kubeeye-manager-role CanModifyWorkloads
ClusterRole vpnkit-controller CanImpersonateUser
ClusterRole vpnkit-controller CanDeleteResources

## What KubeEye can do

* KubeEye inspects cluster resources according with Kubernetes best practices, to make cluster stable.
* KubeEye can find problems of your cluster control plane, including kube-apiserver/kube-controller-manager/etcd, etc.
* KubeEye helps you detect all kinds of cluster nodes problems, including memory/cpu/disk pressure, unexpected kernel error logs, etc.

## Checklist

YES/NO | CHECK ITEM | Description | Level
—|—|—|—

✅

| PrivilegeEscalationAllowed | Privilege [escalation]( “escalation” ) is allowed | danger

✅

| CanImpersonateUser | The role/clusterrole can impersonate other user | warning

✅

| CanModifyResources | The role/clusterrole can delete kubernetes resources | warning

✅

| CanModifyWorkloads | The role/clusterrole can modify kubernetes workloads | warning

✅

| NoCPULimits | The resource does not set limits of CPU in containers.resources | danger

✅

| NoCPURequests | The resource does not set requests of CPU in containers.resources | danger

✅

| HighRiskCapabilities | Have high-Risk options in capabilities such as ALL/SYS_ADMIN/NET_ADMIN | danger

✅

| HostIPCAllowed | HostIPC Set to true | danger

✅

| HostNetworkAllowed | HostNetwork Set to true | danger

✅

| HostPIDAllowed | HostPID Set to true | danger

✅

| HostPortAllowed | HostPort Set to true | danger

✅

| ImagePullPolicyNotAlways | Image pull policy not always | warning

✅

| ImageTagIsLatest | The image tag is latest | warning

✅

| ImageTagMiss | The image tag do not declare | danger

✅

| InsecureCapabilities | Have insecure options in capabilities such as KILL/SYS_CHROOT/CHOWN | danger

✅

| NoLivenessProbe | The resource does not set livenessProbe | warning

✅

| NoMemoryLimits | The resource does not set limits of memory in containers.resources | danger

✅

| NoMemoryRequests | The resource does not set requests of memory in containers.resources | danger

✅

| NoPriorityClassName | The resource does not set priorityClassName | ignore

✅

| PrivilegedAllowed | Running a pod in a privileged mode means that the pod can access the host’s resources and kernel capabilities | danger

✅

| NoReadinessProbe | The resource does not set readinessProbe | warning

✅

| NotReadOnlyRootFilesystem | The resource does not set readOnlyRootFilesystem to true | warning

✅

| NotRunAsNonRoot | The resource does not set runAsNonRoot to true, maybe executed run as a root account | warning

✅

| CertificateExpiredPeriod | Certificate expiration date less than 30 days | danger

✅

| EventAudit | Event audit | warning

✅

| NodeStatus | node status audit | warning

✅

| DockerStatus | docker status audit | warning

✅

| KubeletStatus | kubelet status audit | warning

## Add your own inspection rules

### Add custom OPA rules

* create a [directory]( “directory” ) for OPA rules

mkdir opa

* Add custom OPA rules files

> Note: the OPA rule for workloads, package name must be _kubeeye_workloads_rego_ for RBAC, package name must be _kubeeye_RBAC_rego_ for nodes, package name must be _kubeeye_nodes_rego_

* Save the following rules to rule file such as _imageRegistryRule.rego_ to check the image registry address complies with rules.

package kubeeye_workloads_rego

deny[msg] {
resource := input
type := resource.Object.kind
resourcename := resource.Object.metadata.name
resourcenamespace := resource.Object.metadata.namespace
workloadsType := {“Deployment”,”ReplicaSet”,”DaemonSet”,”StatefulSet”,”Job”}
workloadsType[type]

not workloadsImageRegistryRule(resource)

msg := {
“Name”: sprintf(“%v”, [resourcename]),
“Namespace”: sprintf(“%v”, [resourcenamespace]),
“Type”: sprintf(“%v”, [type]),
“Message”: “ImageRegistryNotmyregistry”
}
}

workloadsImageRegistryRule(resource) {
regex.match(“^myregistry.public.kubesphere/basic/.+”, resource.Object.spec.template.spec.containers[_].image)
}

* Run KubeEye with custom rules

> Note: Specify the path then Kubeeye will read all files in the directory that end with _.rego_.

root:# kubeeye audit -p ./opa
NAMESPACE NAME KIND MESSAGE
default nginx1 Deployment [ImageRegistryNotmyregistry NotReadOnlyRootFilesystem NotRunAsNonRoot]
default nginx11 Deployment [ImageRegistryNotmyregistry PrivilegeEscalationAllowed HighRiskCapabilities HostIPCAllowed HostPortAllowed ImagePullPolicyNotAlways ImageTagIsLatest InsecureCapabilities NoPriorityClassName PrivilegedAllowed NotReadOnlyRootFilesystem NotRunAsNonRoot]
default nginx111 Deployment [ImageRegistryNotmyregistry NoCPULimits NoCPURequests ImageTagMiss NoLivenessProbe NoMemoryLimits NoMemoryRequests NoPriorityClassName NotReadOnlyRootFilesystem NoReadinessProbe NotRunAsNonRoot]

### Add custom NPD rules

* edit configmap

kubectl edit ConfigMap node-problem-detector-config -n kube-system

* restart NPD deployment

kubectl rollout restart DaemonSet node-problem-detector -n kube-system

## KubeEye Operator

### What is KubeEye Operator

KubeEye Operator is an inspection platform for Kubernetes, manage KubeEye by operator and generate inspection result.

### What KubeEye Operator can do

* KubeEye Operator provides [management]( “management” ) functions through web page.
* KubeEye Operator recode inspection results by CR, can view and compare cluster inspection results by web page.
* KubeEye Operator provides more plugins.
* KubeEye Operator provides more detailed modification suggestions.

### deploy Kubeeye

kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye.yaml
kubectl apply -f https://raw.githubusercontent.com/kubesphere/kubeeye/main/deploy/kubeeye_insights.yaml

### get the inspection results

kubectl get clusterinsight -o yaml

apiVersion: v1
items:
– apiVersion: kubeeye.kubesphere.io/v1alpha1
kind: ClusterInsight
metadata:
name: clusterinsight-sample
namespace: default
spec:
auditPeriod: 24h
status:
auditResults:
auditResults:
– resourcesType: Node
resultInfos:
– namespace: “”
resourceInfos:
– items:
– level: warning
message: KubeletHasNoSufficientMemory
reason: kubelet has no sufficient memory available
– level: warning
message: KubeletHasNoSufficientPID
reason: kubelet has no sufficient PID available
– level: warning
message: KubeletHasDiskPressure
reason: kubelet has disk pressure
name: kubeeyeNode

## Documents

* [RoadMap]( “RoadMap” )
* [FAQ]( “FAQ” )

**[Download Kubeeye]( “Download Kubeeye” )**Read More