Building Infrastructure Observability at Monesize

Infrastructure monitoring became a priority at Monesize after we experienced firsthand how difficult it can be to diagnose production problems without a centralized view of system health. We have individual servers running the company's services, application logs available through PM2, and AWS providing infrastructure status. However, these sources did not give us one place to understand what was happening across the infrastructure when something went wrong.

The problem became clear during a production incident in which I could no longer access the backend over SSH, even though the EC2 instance still appeared as running in the AWS console. More critically, all the applications running across the infrastructure became inaccessible through their web interfaces, and the monitoring endpoints also stopped being accessible. This was not an isolated backend issue. The outage affected the applications themselves, while the backend logs also exposed database and Redis connectivity problems.

The whole incident lasted for over an hour and the services started coming up after the affected instance was rebooted from the AWS Console. Our central SIEM was checked thoroughly but no signal of a possible DDoS attack was found. We couldn't tell exactly what went wrong.

So, I decided to address this gap by building an infrastructure observability stack around Prometheus, Node Exporter and Grafana, hosted on the existing SIEM monitoring server. I wanted a practical solution that could collect system metrics centrally, display them through dashboards and fit into the security boundaries we had already established in AWS Security Groups.

This post documents the architecture, implementation, configuration and lessons from that work.

1. The incident that changed my approach

The immediate motivation came from a production infrastructure incident. The AWS console reported that the production EC2 instance was running, but I could not establish an SSH connection to the backend machine. I could access other instances, which made the problem more difficult to isolate.

Application logs revealed additional symptoms. Prisma reported that the application could not find the expected public.User table in the database. The Node.js backend also reported Redis connection timeouts through ioredis. These errors pointed to different layers of the system, so I had to investigate the application, database state, network connectivity and process management rather than assume that one failure explained everything.

I inspected PM2, reviewed application logs, restarted the affected backend process and investigated the supporting services. The experience reinforced an important operational distinction: AWS reporting an instance as running does not establish that the applications and dependencies inside it are healthy.

The incident did not, by itself, prove that resource exhaustion caused the SSH problem or the application errors. I needed better evidence to make that kind of diagnosis confidently. Without historical CPU, memory, disk and network metrics, I had limited ability to compare system conditions before, during and after the incident.

I also wanted to avoid relying entirely on manual investigation after an issue had already affected production. I needed a consistent way to answer basic questions:

  • Is the host reachable by the monitoring system?
  • Is CPU usage unusually high?
  • Is available memory falling?
  • Is a filesystem approaching capacity?
  • Has network traffic changed significantly?
  • Did system resource usage change around the time an application began failing?

I therefore treated centralized infrastructure monitoring as an operational requirement rather than a future enhancement. I wanted to establish a baseline of system health before the next incident and make troubleshooting more evidence-driven.

2. Why I chose Prometheus, Node Exporter and Grafana

I wanted to build on the infrastructure I already operated instead of introducing a collection of unrelated monitoring products. I selected three components, each with a specific responsibility.

Prometheus for metrics collection

Prometheus collects time-series metrics by periodically querying HTTP endpoints exposed by monitored systems. It stores those measurements and lets me query them over time.

I use Prometheus as the central metrics collector. It scrapes its own metrics and the Node Exporter endpoints on the monitored Linux hosts. I configured a global scrape interval of 15 seconds, which gives me a useful view of short-term resource changes without requiring continuous manual checks.

Node Exporter for Linux system metrics

Node Exporter exposes hardware and operating system metrics through an HTTP endpoint. It provides information about CPU activity, memory, filesystems, network interfaces and other host-level characteristics.

I installed Node Exporter directly on each Linux machine as a systemd service. This keeps it independent of the application processes it monitors. A Node.js application can fail while Node Exporter continues to expose system metrics, provided the host and exporter remain operational.

Grafana for visualization

Prometheus provides the collection and querying layer, but I wanted dashboards that made the information easier to interpret. I use Grafana to connect to Prometheus and visualize the collected time series.

I imported the Node Exporter Full dashboard, which provides panels for common Linux resource metrics. I also configured the dashboard to expose each monitored host through a job selector, allowing me to move between hosts without maintaining a separate dashboard for every machine.

The separation of responsibilities is deliberate:

  • Node Exporter exposes host metrics.
  • Prometheus collects and stores those metrics.
  • Grafana queries Prometheus and presents the results.

This architecture gives me a straightforward monitoring stack that I can extend as Monesize's infrastructure and application requirements evolve.

3. The architecture

I deployed Prometheus and Grafana on the existing Wazuh monitoring server. Each monitored Linux host runs its own Node Exporter service.

Node Exporter exposes system metrics over HTTP on port 9100. Prometheus periodically scrapes these endpoints and stores the collected metrics as time series. Grafana connects to Prometheus and presents the data through dashboards that let me inspect CPU utilization, memory consumption, disk capacity and network activity across the infrastructure.

I kept the Prometheus web interface bound to localhost, restricted Node Exporter access through AWS security groups, and exposed Grafana through the existing Nginx reverse-proxy setup with HTTPS.

I also kept the observability stack separate from the application processes it monitors. That separation reduces the likelihood that an ordinary application restart will interrupt metrics collection.

The architecture does introduce a shared dependency: Prometheus and Grafana run on the Wazuh host. A failure of that host could interrupt the central monitoring interface, even while the application hosts continue running. The stack improves visibility, but it does not make the monitoring system itself highly available. This was a decision made on constraints as we couldn't just provision a fresh EC2 instance. Our balance sheet was already struggling to catch breath. We no want make him kpai (lol).

4. Installing Node Exporter on the Linux hosts

I installed Node Exporter on the monitored hosts before configuring Prometheus to scrape them. I used version 1.12.1 for this deployment and installed the binary under /usr/local/bin.

The installation requires the correct binary for each machine's operating system and processor architecture. Our x86-64 Ubuntu hosts use the AMD64 release, while the ARM64 host requires the ARM64 release. The operating system alone does not determine which binary to download. I verified the processor architecture first.

Verify the operating system and architecture

cat /etc/os-release
uname -m

The uname -m command reports the machine architecture. Typical outputs include x86_64 for AMD64 systems and aarch64 for ARM64 systems.

I, then, downloaded the matching Node Exporter archive from the official Prometheus GitHub releases page.

Download and install the correct binary

For an AMD64 Linux host, I used:

cd /tmp
curl -fLO https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz
tar xzf node_exporter-1.12.1.linux-amd64.tar.gz
sudo install \
  node_exporter-1.12.1.linux-amd64/node_exporter \
  /usr/local/bin/node_exporter

For the ARM64 host, I used the corresponding ARM64 release:

cd /tmp
curl -fLO https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-arm64.tar.gz
tar xzf node_exporter-1.12.1.linux-arm64.tar.gz
sudo install \
  node_exporter-1.12.1.linux-arm64/node_exporter \
  /usr/local/bin/node_exporter

The ARM64 distinction mattered during implementation. I initially installed the AMD64 binary on the ARM64 machine. The file existed at the expected path, but Linux returned an Exec format error because the binary targeted a different processor architecture.

I confirmed the mismatch with:

uname -m
file /usr/local/bin/node_exporter

The correction was to replace the incompatible binary with the ARM64 build. This was a useful reminder that a successful download and file installation do not establish that an executable can run on the target machine.

Create a dedicated service account

I created a system user for Node Exporter rather than running it as root:

sudo useradd \
  --no-create-home \
  --shell /usr/sbin/nologin \
  node_exporter

The account does not need an interactive login. Running the exporter under a dedicated service account also avoids granting it unnecessary root privileges.

Create the systemd service

I created /etc/systemd/system/node_exporter.service:

[Unit]
Description=Prometheus Node Exporter
After=network-online.target
Wants=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target

I used this service definition on the Ubuntu application hosts. The service manager now handles the exporter's lifecycle and starts it automatically during boot.

I enabled and started the service:

sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

I checked its status with:

sudo systemctl status node_exporter --no-pager

I looked for active (running) before moving on.

Verify the metrics endpoint

Node Exporter listens on TCP port 9100 by default. I verified that the exporter exposed metrics:

curl http://localhost:9100/metrics

To inspect a smaller sample, I used:

curl -s http://localhost:9100/metrics \
  | grep -E '^node_(cpu|memory|filesystem|network)_'

The output includes metrics such as node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes and network counters. These provide the raw measurements that Prometheus collects.

I repeated the installation and verification process on the relevant Linux hosts, selecting the correct binary architecture for each one.

5. Configuring network access securely

Installing an exporter is only part of the job. Prometheus must be able to reach its metrics endpoint, and that endpoint must not become an unrestricted public service.

I used AWS security groups to control access to TCP port 9100. I allowed the monitoring server as an approved source instead of opening the port to every IP address on the internet.

This is important because Node Exporter exposes infrastructure information that I do not want to make publicly accessible. The endpoint does not require a public-facing interface for this architecture. Prometheus only needs a network path to the exporter.

I applied the same principle to Prometheus itself. Its web interface listens on 127.0.0.1:9090, so I did not expose port 9090 directly to the internet.

Grafana uses a different access model. I wanted to access dashboards through the company's existing domain and HTTPS reverse proxy, rather than expose the application server's internal port directly.

These controls create three distinct boundaries:

  1. AWS security groups restrict which systems can reach Node Exporter.
  2. Prometheus listens on localhost instead of a public interface.
  3. Nginx provides the external HTTPS entry point for Grafana.

The design keeps the monitoring data path separate from the browser-facing dashboard path. Grafana can query Prometheus locally while users access Grafana through HTTPS.

6. Installing and configuring Prometheus

I installed Prometheus on the Wazuh monitoring server using the official Linux release archive. I used Prometheus v3.15.0 for this deployment.

I downloaded and extracted the archive, then installed the executable and configuration utilities:

cd /tmp
curl -fLO https://github.com/prometheus/prometheus/releases/download/v3.15.0/prometheus-3.15.0.linux-amd64.tar.gz
tar xzf prometheus-3.15.0.linux-amd64.tar.gz
sudo install \
  prometheus-3.15.0.linux-amd64/prometheus \
  /usr/local/bin/prometheus
sudo install \
  prometheus-3.15.0.linux-amd64/promtool \
  /usr/local/bin/promtool

I created a dedicated system account and the directories Prometheus needs:

sudo useradd \
  --no-create-home \
  --shell /usr/sbin/nologin \
  prometheus

sudo mkdir -p /etc/prometheus
sudo mkdir -p /var/lib/prometheus

sudo chown -R prometheus:prometheus \
  /etc/prometheus /var/lib/prometheus

I used /etc/prometheus/prometheus.yml for the configuration and /var/lib/prometheus for the time-series database.

Configure the scrape jobs

Prometheus uses scrape jobs to define the endpoints it queries. Each job groups targets under a name that becomes a label in the collected metrics. This allows me to distinguish one host from another in Grafana and PromQL queries.

My configuration follows this structure:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "vm1"
    static_configs:
      - targets: ["<VM1_PRIVATE_IP>:9100"]

  - job_name: "vm2"
    static_configs:
      - targets: ["<VM2_REACHABLE_ADDRESS>:9100"]

  - job_name: "vm3"
    static_configs:
      - targets: ["<VM3_REACHABLE_ADDRESS>:9100"]

  - Every other host followed same syntax

The address placeholders represent the actual reachable addresses configured in my environment. I have omitted them here rather than publish internal network details. I gave each host a descriptive job name. This makes the data easier to query and lets me select individual hosts in the Grafana dashboard.

The prometheus job monitors Prometheus itself, while the other jobs collect system metrics from Node Exporter.

Create the systemd service

I created /etc/systemd/system/prometheus.service:

[Unit]
Description=Prometheus Monitoring
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus

Restart=on-failure

[Install]
WantedBy=multi-user.target

This configuration runs Prometheus as its dedicated user, reads the scrape configuration from /etc/prometheus/prometheus.yml and stores metrics under /var/lib/prometheus.

I enabled the service:

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

I then checked the service:

sudo systemctl status prometheus --no-pager

Validate configuration changes

Before restarting Prometheus after changing its configuration, I used promtool to validate the YAML:

sudo /usr/local/bin/promtool check config \
  /etc/prometheus/prometheus.yml

The command returned:

SUCCESS: prometheus.yml is valid prometheus config file syntax

This caught configuration errors before they could interfere with the running service. I restarted Prometheus after adding the new targets:

sudo systemctl restart prometheus

I checked that the service returned to active (running).

One implementation detail worth documenting: a plain systemctl reload prometheus did not work with my systemd unit because I had not configured a reload action. Restarting the service applied the configuration successfully. Prometheus also supports other reload mechanisms when an operator enables and configures them.

7. Installing Grafana and connecting Prometheus

With metrics collection working, I installed Grafana on the same monitoring server. I used the official Grafana RPM repository and installed the grafana package.

I started and enabled the service:

sudo systemctl enable --now grafana-server

I checked its status:

sudo systemctl status grafana-server --no-pager

Bind Grafana to the internal interface

I edited /etc/grafana/grafana.ini and configured the server settings:

[server]
http_addr = <MONITORING_SERVER_PRIVATE_IP>
http_port = 3000
root_url = https://grafana.monesize.com/

I replaced the address placeholder with the monitoring server's private IP. Binding Grafana to the internal interface means that Nginx can reach it without requiring me to expose its port directly to the public internet.

I restarted Grafana after changing the configuration:

sudo systemctl restart grafana-server

I checked the local login endpoint:

curl -I http://localhost:3000/login

A successful HTTP response confirmed that Grafana was serving the login page.

Provision the Prometheus data source

I wanted Grafana to connect to Prometheus without requiring a manual data-source configuration after every deployment.

I created /etc/grafana/provisioning/datasources/prometheus.yml:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://127.0.0.1:9090
    isDefault: true
    editable: true

The url points to the Prometheus instance on localhost. The proxy access mode tells Grafana to query Prometheus from the server side rather than ask the user's browser to connect to the Prometheus endpoint directly. Grafana supports YAML-based data-source provisioning, which makes this configuration repeatable and easier to manage.

I restarted Grafana: sudo systemctl restart grafana-server

I then logged in to Grafana and confirmed that the Prometheus data source was available.

8. Publishing Grafana through Nginx and HTTPS

I already used Nginx as a reverse proxy for services on the monitoring server. I extended that configuration to serve Grafana through its own subdomain.

I created /etc/nginx/conf.d/grafana.conf:

server {
    listen 80;
    server_name grafana.monesize.com;

    location / {
        proxy_pass http://<MONITORING_SERVER_PRIVATE_IP>:3000;

        proxy_http_version 1.1;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

I replaced the address placeholder with the monitoring server's private IP. The forwarded headers preserve request information, while the upgrade headers support WebSocket connections.

I validated the Nginx configuration: sudo nginx -t

After the test passed, I reloaded Nginx: sudo systemctl reload nginx

I then used Certbot to issue and configure the HTTPS certificate:

sudo certbot --nginx -d grafana.monesize.com

Once certificate issuance and Nginx configuration succeeded, I accessed Grafana through: https://grafana.monesize.com

This gave me browser access to the dashboards without exposing Grafana's application port directly.

9. Building the dashboard

I imported the Node Exporter Full dashboard, identified by dashboard ID 1860, from Grafana's dashboard catalogue. The dashboard provides a practical starting point for viewing common Linux system metrics. Rather than build every panel from scratch, I used it to visualize the metrics that Node Exporter already exposed.

However, importing a dashboard does not automatically make it suitable for every environment. The dashboard needs to query the correct data source and discover the jobs and instances that Prometheus actually collects.

Fixing the job selector

Initially, the dashboard's Job dropdown displayed only vm1, even though I had added other targets to Prometheus.

I investigated the problem in Grafana and refreshed the dashboard configuration. After the refresh, the dropdown displayed the other jobs. The issue was with the dashboard's available variable values, not the underlying metrics collection. The targets were already reporting successfully.

This distinction matters because a dashboard can appear incomplete even when Prometheus is working correctly. I verified the actual data before changing more configuration.

10. Verifying that the complete stack works

I used a layered verification process instead of relying on a single green status indicator.

Check the Prometheus targets

In Grafana Explore, I ran: up

The up metric reports whether Prometheus successfully scraped a target during its most recent scrape. A value of 1 means the scrape succeeded, while 0 means it failed. I confirmed that the Prometheus server and all monitored Node Exporter targets returned 1.

I also queried individual jobs using this query: up{job="vm1"}

These checks confirmed that Prometheus could reach each exporter and collect metrics.

Check CPU utilization

Node Exporter exposes cumulative CPU time counters. I can calculate the proportion of CPU time spent outside the idle mode over a recent interval:

100 * (
  1 - avg by (instance) (
    rate(node_cpu_seconds_total{mode="idle"}[5m])
  )
)

This gives me an estimate of total CPU utilization per instance. The five-minute window smooths short fluctuations and makes sustained load easier to interpret.

Check available memory

I can inspect available memory as a percentage of total memory:

100 *
node_memory_MemAvailable_bytes
/
node_memory_MemTotal_bytes

This helps me identify hosts with declining memory headroom.

Check filesystem capacity

To examine the percentage of filesystem space used:

100 * (
  1 -
  node_filesystem_avail_bytes{
    fstype!~"tmpfs|overlay",
    mountpoint!=""
  }
  /
  node_filesystem_size_bytes{
    fstype!~"tmpfs|overlay",
    mountpoint!=""
  }
)

I can use the resulting time series to identify filesystems approaching capacity. For operational use, I would also filter pseudo-filesystems and apply mountpoint-specific thresholds to reduce noise.

Check network traffic

To inspect incoming network traffic by interface: 

rate(node_network_receive_bytes_total[5m])

For outgoing traffic:

rate(node_network_transmit_bytes_total[5m])

These queries calculate byte rates rather than displaying the raw cumulative counters. That makes them more useful for examining changes in network activity.

Together, these queries provide a starting point for investigating resource pressure. I still need application logs and service-level metrics to establish why a particular application request failed.

11. Memory management on the monitoring server

Running observability tools alongside Wazuh meant I had to pay attention to the monitoring server's own resource consumption.

During implementation, I observed that the Wazuh Indexer Java process consumed a substantial portion of the server's memory. Grafana also used noticeable memory after startup. Prometheus itself consumed comparatively little memory during the initial checks.

I inspected memory usage with:

free -h

To identify the largest processes, I ran:

ps aux --sort=-%mem | head -15

I also examined the Wazuh Indexer JVM heap settings in:

/etc/wazuh-indexer/jvm.options

The original heap configuration used approximately 3.9 GB for both the minimum and maximum heap. I reduced both values to approximately 2.9 GB and restarted the indexer. I then checked the service status and reviewed available memory again.

I also configured a 1 GB swap file as additional protection against sudden memory pressure. Swap does not replace RAM, and heavy swapping can degrade performance. I treated it as a fallback rather than a reason to allocate more memory than the server could support.

This exercise reinforced another useful principle: the monitoring system needs monitoring too. A stack that collects metrics from other hosts but runs out of memory itself can lose the visibility it was designed to provide.

12. What this implementation gives Monesize

The deployment provides a centralized view of Linux infrastructure health through a stack that I can operate and extend myself. I can now:

  • Compare CPU utilization across hosts.
  • Review available memory and identify declining headroom.
  • Track filesystem capacity.
  • Inspect network traffic.
  • Examine historical resource metrics instead of relying exclusively on current state.
  • Verify whether Prometheus can reach each monitored exporter.
  • Move between hosts through one Grafana dashboard.
  • Access the dashboard over HTTPS without exposing the Prometheus interface publicly.

The most important improvement is the ability to establish a resource baseline. When another incident occurs, I will have time-series data that I can compare against the incident window.

That evidence will not automatically explain every failure. A healthy CPU graph cannot prove that a database schema is correct, and a successful Node Exporter scrape cannot establish that an application endpoint returns valid responses. However, the metrics can help me rule out certain causes, identify resource pressure and narrow the scope of an investigation.

13. What this stack does not yet cover

I consider this implementation an infrastructure observability foundation, not a complete application observability platform. Node Exporter reports host-level metrics. It does not automatically instrument our applications, measure individual API endpoint latency or trace a request through the ORM and Redis.

The stack also does not yet provide a complete alerting and notification workflow. Grafana dashboards make metrics visible, but visibility alone does not guarantee that we will learn about a problem before a customer reports it.

Our next layer of observability will need to address three areas:

Application metrics: to measure request volume, response latency, HTTP error rates and dependency failures directly from the applications.

Centralized logs: to correlate application errors with infrastructure metrics without connecting to individual hosts and manually searching local log files.

Distributed traces: to follow requests through the application and its dependencies to identify where latency or failures occur.

OpenTelemetry provides an instrumentation framework for collecting application telemetry. We can introduce it into the applications' services and extend the existing stack as those requirements become more important. We do not need to replace Prometheus or Grafana to take that step.

We also need to establish alert rules, delivery channels, sensible thresholds and a process for responding to alerts. Until we implement and test those pieces, we will not treat the stack as a complete proactive incident-response system.

Well Well...

I built infrastructure observability at Monesize because the production incident exposed a gap in how we understood and monitored the systems we operated. We could inspect individual logs and check AWS instance status, but we lacked a centralized view of system resource health and its changes over time.

Prometheus, Node Exporter and Grafana gave us that foundation without requiring a wholesale change to the existing infrastructure. I installed exporters on the Linux hosts, restricted access through AWS security groups, configured Prometheus to scrape the metrics, provisioned Grafana's data source and published the dashboard through Nginx and HTTPS.

The resulting stack gives us a practical way to see infrastructure health in one place. It also gives us a stronger starting point for troubleshooting because we can use historical measurements to investigate what happened around an incident rather than relying entirely on memory and isolated logs.

I do not consider the work finished forever. I consider the immediate infrastructure visibility gap addressed. We can now build on that foundation with application metrics, centralized logs, distributed traces and alerting when the operational need justifies each addition.

For me, the central lesson is straightforward: we cannot manage infrastructure reliably when we lack visibility into its behaviour. I want Monesize's operational decisions to rely on evidence, and this observability stack is one practical step toward that standard.

Comments

Popular Posts

Exploiting MS17-010 EternalBlue: SMB Flaw to SYSTEM Access

RecruitX: From Reconnaissance to Remote Code Execution

God Never Wrote a Book: A Nigerian Agnostic's Case