Cyber Post

Auditd in Kubernetes (Part 3)

Dec 12, 2024 by J

Updated Dec 2024 2 tags
What to expect

Real challenges, actual solutions, and what I learned in the process.

This blog post talks about how to install and setup Auditd in Kubernetes.

Introduction

I followed a medium guide 1 on the basic installation of Auditd on Ubuntu. You can edit the config file as you deem fit. The tricky part is how to configure promtail to send these logs back to the loki server - which i cant find any guides for. After installing auditd packages, you’ll have to get an auditd config file. I used a popular config file by Neo23x0 on github.

sudo apt-get install auditd audispd-plugins
wget -O audit.rules https://raw.githubusercontent.com/Neo23x0/auditd/master/audit.rules
cp audit.rules /etc/audit/rules.d/
systemctl restart auditd.service
systemctl enable auditd.service

With that, auditd should be running. You should find the log files in /var/log/audit/*. If you want to change the path where the logs are stored, you can edit the conf file with the command below. The parameter is called log_file.

sudo vi /etc/audit/auditd.conf

To configure the promtail agent to send your log file to loki, you will have to make edits to promtail.yaml which can be find in the promtail pods. Use kubectl get pods -A and find the promtail pod and exec into it.

kubectl exec -it -n prometheus-grafana loki-promtail-bgzrh -- sh
ls
# bin boot dev etc home lib	lib64 media mnt opt	proc root run sbin srv sys tmp usr var
cd /etc/promtail
ls
#promtail.yaml
cat promtail.yaml
promtail.yaml

Copy down the output and save it into a promtail.yaml file. Within that file, you have to include the path of the auditd logs that you want promtail to forward. Your edit should be something like this.

scrape_configs:
- job_name: audit-logs
static_configs:
- targets:
- localhost
labels:
job: audit-logs
__path__: /var/log/audit/*
Editing daemonset config

Once that is done, download the daemonset config and make sure the path where auditd logs are stored is mounted.

kubectl get daemonset loki-promtail -n prometheus-grafana -o yaml > loki-promtail.yaml
kubectl apply -f loki-promtail.yaml

### You have to add this in your loki-promtail.yaml before running the helm upgrade command
- mountPath: /var/log/audit
name: audit-logs
readOnly: true

helm upgrade loki --namespace=prometheus-grafana grafana/loki-stack -f loki-stack-value.yaml
The most important part

You need to make sure that loki-promtail uses this promtail.yaml file for all nodes (in the event that auditd is present in all nodes).

kubectl get secret loki-promtail -n prometheus-grafana -o yaml > promtail-secret.yaml

Within your promtail-secret.yaml file, you should see a data: field with some encoded code. That is the base64 encoded values for the promtail.yaml file that you edited previously. Copy paste your entire promtail.yaml into any b64 encoding tool like CyberChef and paste it into the promtail-secret.yaml file. You can decode my b64 string by clicking here 🔗 to see my promtail.yaml configuration file. Your final version of the .yaml file should be something like this:

data:
promtail.yaml: c2VydmVyOg0KICBsb2dfbGV2ZWw6IGluZm8NCiAgbG9nX2Zvcm1hdDogbG9nZm10DQogIGh0dHBfbGlzdGVuX3BvcnQ6IDMxMDENCiAgDQoNCmNsaWVudHM6DQogIC0gdXJsOiBodHRwOi8vbG9raTozMTAwL2xva2kvYXBpL3YxL3B1c2gNCg0KcG9zaXRpb25zOg0KICBmaWxlbmFtZTogL3J1bi9wcm9tdGFpbC9wb3NpdGlvbnMueWFtbA0KDQpzY3JhcGVfY29uZmlnczoNCiAgLSBqb2JfbmFtZTogYXVkaXQtbG9ncw0KICAgIHN0YXRpY19jb25maWdzOg0KICAgIC0gdGFyZ2V0czoNCiAgICAgICAgLSBsb2NhbGhvc3QNCiAgICAgIGxhYmVsczoNCiAgICAgICAgam9iOiBhdWRpdC1sb2dzDQogICAgICAgIF9fcGF0aF9fOiAvdmFyL2xvZy9hdWRpdC8qIA0KICAtIGpvYl9uYW1lOiBrdWJlcm5ldGVzLXBvZHMNCiAgICBwaXBlbGluZV9zdGFnZXM6DQogICAgICAtIGNyaToge30NCiAgICBrdWJlcm5ldGVzX3NkX2NvbmZpZ3M6DQogICAgICAtIHJvbGU6IHBvZA0KICAgIHJlbGFiZWxfY29uZmlnczoNCiAgICAgIC0gc291cmNlX2xhYmVsczoNCiAgICAgICAgICAtIF9fbWV0YV9rdWJlcm5ldGVzX3BvZF9jb250cm9sbGVyX25hbWUNCiAgICAgICAgcmVnZXg6IChbMC05YS16LS5dKz8pKC1bMC05YS1mXXs4LDEwfSk/DQogICAgICAgIGFjdGlvbjogcmVwbGFjZQ0KICAgICAgICB0YXJnZXRfbGFiZWw6IF9fdG1wX2NvbnRyb2xsZXJfbmFtZQ0KICAgICAgLSBzb3VyY2VfbGFiZWxzOg0KICAgICAgICAgIC0gX19tZXRhX2t1YmVybmV0ZXNfcG9kX2xhYmVsX2FwcF9rdWJlcm5ldGVzX2lvX25hbWUNCiAgICAgICAgICAtIF9fbWV0YV9rdWJlcm5ldGVzX3BvZF9sYWJlbF9hcHANCiAgICAgICAgICAtIF9fdG1wX2NvbnRyb2xsZXJfbmFtZQ0KICAgICAgICAgIC0gX19tZXRhX2t1YmVybmV0ZXNfcG9kX25hbWUNCiAgICAgICAgcmVnZXg6IF47KihbXjtdKykoOy4qKT8kDQogICAgICAgIGFjdGlvbjogcmVwbGFjZQ0KICAgICAgICB0YXJnZXRfbGFiZWw6IGFwcA0KICAgICAgLSBzb3VyY2VfbGFiZWxzOg0KICAgICAgICAgIC0gX19tZXRhX2t1YmVybmV0ZXNfcG9kX2xhYmVsX2FwcF9rdWJlcm5ldGVzX2lvX2luc3RhbmNlDQogICAgICAgICAgLSBfX21ldGFfa3ViZXJuZXRlc19wb2RfbGFiZWxfaW5zdGFuY2UNCiAgICAgICAgcmVnZXg6IF47KihbXjtdKykoOy4qKT8kDQogICAgICAgIGFjdGlvbjogcmVwbGFjZQ0KICAgICAgICB0YXJnZXRfbGFiZWw6IGluc3RhbmNlDQogICAgICAtIHNvdXJjZV9sYWJlbHM6DQogICAgICAgICAgLSBfX21ldGFfa3ViZXJuZXRlc19wb2RfbGFiZWxfYXBwX2t1YmVybmV0ZXNfaW9fY29tcG9uZW50DQogICAgICAgICAgLSBfX21ldGFfa3ViZXJuZXRlc19wb2RfbGFiZWxfY29tcG9uZW50DQogICAgICAgIHJlZ2V4OiBeOyooW147XSspKDsuKik/JA0KICAgICAgICBhY3Rpb246IHJlcGxhY2UNCiAgICAgICAgdGFyZ2V0X2xhYmVsOiBjb21wb25lbnQNCiAgICAgIC0gYWN0aW9uOiByZXBsYWNlDQogICAgICAgIHNvdXJjZV9sYWJlbHM6DQogICAgICAgIC0gX19tZXRhX2t1YmVybmV0ZXNfcG9kX25vZGVfbmFtZQ0KICAgICAgICB0YXJnZXRfbGFiZWw6IG5vZGVfbmFtZQ0KICAgICAgLSBhY3Rpb246IHJlcGxhY2UNCiAgICAgICAgc291cmNlX2xhYmVsczoNCiAgICAgICAgLSBfX21ldGFfa3ViZXJuZXRlc19uYW1lc3BhY2UNCiAgICAgICAgdGFyZ2V0X2xhYmVsOiBuYW1lc3BhY2UNCiAgICAgIC0gYWN0aW9uOiByZXBsYWNlDQogICAgICAgIHJlcGxhY2VtZW50OiAkMQ0KICAgICAgICBzZXBhcmF0b3I6IC8NCiAgICAgICAgc291cmNlX2xhYmVsczoNCiAgICAgICAgLSBuYW1lc3BhY2UNCiAgICAgICAgLSBhcHANCiAgICAgICAgdGFyZ2V0X2xhYmVsOiBqb2INCiAgICAgIC0gYWN0aW9uOiByZXBsYWNlDQogICAgICAgIHNvdXJjZV9sYWJlbHM6DQogICAgICAgIC0gX19tZXRhX2t1YmVybmV0ZXNfcG9kX25hbWUNCiAgICAgICAgdGFyZ2V0X2xhYmVsOiBwb2QNCiAgICAgIC0gYWN0aW9uOiByZXBsYWNlDQogICAgICAgIHNvdXJjZV9sYWJlbHM6DQogICAgICAgIC0gX19tZXRhX2t1YmVybmV0ZXNfcG9kX2NvbnRhaW5lcl9uYW1lDQogICAgICAgIHRhcmdldF9sYWJlbDogY29udGFpbmVyDQogICAgICAtIGFjdGlvbjogcmVwbGFjZQ0KICAgICAgICByZXBsYWNlbWVudDogL3Zhci9sb2cvcG9kcy8qJDEvKi5sb2cNCiAgICAgICAgc2VwYXJhdG9yOiAvDQogICAgICAgIHNvdXJjZV9sYWJlbHM6DQogICAgICAgIC0gX19tZXRhX2t1YmVybmV0ZXNfcG9kX3VpZA0KICAgICAgICAtIF9fbWV0YV9rdWJlcm5ldGVzX3BvZF9jb250YWluZXJfbmFtZQ0KICAgICAgICB0YXJnZXRfbGFiZWw6IF9fcGF0aF9fDQogICAgICAtIGFjdGlvbjogcmVwbGFjZQ0KICAgICAgICByZWdleDogdHJ1ZS8oLiopDQogICAgICAgIHJlcGxhY2VtZW50OiAvdmFyL2xvZy9wb2RzLyokMS8qLmxvZw0KICAgICAgICBzZXBhcmF0b3I6IC8NCiAgICAgICAgc291cmNlX2xhYmVsczoNCiAgICAgICAgLSBfX21ldGFfa3ViZXJuZXRlc19wb2RfYW5ub3RhdGlvbnByZXNlbnRfa3ViZXJuZXRlc19pb19jb25maWdfaGFzaA0KICAgICAgICAtIF9fbWV0YV9rdWJlcm5ldGVzX3BvZF9hbm5vdGF0aW9uX2t1YmVybmV0ZXNfaW9fY29uZmlnX2hhc2gNCiAgICAgICAgLSBfX21ldGFfa3ViZXJuZXRlc19wb2RfY29udGFpbmVyX25hbWUNCiAgICAgICAgdGFyZ2V0X2xhYmVsOiBfX3BhdGhfXw0KDQpsaW1pdHNfY29uZmlnOg0KICANCnRyYWNpbmc6DQogIGVuYWJsZWQ6IGZhbHNl
kind: Secret

With that completed, apply the changes and thats all. You should be able to view and query your auditd logs from Grafana.

kubectl apply -f promtail-secret.yaml
Validation from Grafana dashboard