Tools

Tools: SigNoz Setup Guide - Monitor Server (2026)

2026-04-18 0 views admin
Tools: SigNoz Setup Guide - Monitor Server (2026)

Environment

1. Install SigNoz

2. Fix ClickHouse — Disable simdjson (REQUIRED for CPUs without AVX2)

Why this happens

3. Disable ClickHouse System Logs (REQUIRED — prevent disk exhaustion)

4. Configure SMTP for Email Alerts

5. Start SigNoz

6. Setup Host Metrics Monitoring (Infrastructure Tab)

6.1 Create config file on the main server

6.2 Run the container

7. Create Disk Usage Alert (> 80%)

8. Routine Maintenance

9. Troubleshooting

ClickHouse disk full due to system logs

Infrastructure Hosts tab returns 500 error

otel-agent not sending data CPUs without AVX2 will encounter CANNOT_ALLOCATE_MEMORY errors when SigNoz queries metrics via JSONExtractString. Check if your CPU supports AVX2: Edit /root/signoz/deploy/common/clickhouse/users.xml and add <allow_simdjson>0</allow_simdjson> to the default profile: ClickHouse uses the simdjson library for JSON parsing functions (JSONExtractString, etc.). Starting from ClickHouse 24.1+, simdjson requires AVX2 CPU instructions. On CPUs without AVX2, any query using JSONExtractString on the distributed_time_series_v4 table throws: Setting allow_simdjson=0 forces ClickHouse to fall back to rapidjson, which does not require AVX2. By default, ClickHouse writes internal diagnostic logs (trace_log, metric_log, etc.) that can grow to 70GB+ over time. Create /root/signoz/deploy/common/clickhouse/z_log_disable.xml: The z_ prefix ensures this file is loaded last (configs are applied alphabetically), overriding any previous settings. Mount this file into the ClickHouse container in /root/signoz/deploy/docker/docker-compose.yaml: Add the following environment variables to the signoz service in docker-compose.yaml: Gmail note: You must use an App Password, not your regular Gmail password.

Generate one at: https://myaccount.google.com/apppasswords

The App Password is 16 characters — remove all spaces before using it. Important: The correct variable names use the SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__ prefix with double underscores. Using SIGNOZ_ALERTMANAGER_SMTP_* (without SIGNOZ_GLOBAL) will not work. Verify all containers are healthy: Verify ClickHouse system logs are disabled: Run an otel-collector agent on the main application server: Replace <SIGNOZ_MONITOR_SERVER_IP> with your SigNoz monitor server IP. After a few minutes, the host should appear in SigNoz UI → Infrastructure → Hosts. Go to SigNoz UI → Alerts → New Alert → Metric Based Alert: Query B (total disk — all states): Formula F1: A / B * 100 Root cause: CPU does not support AVX2 instructions. Fix: Add <allow_simdjson>0</allow_simdjson> to ClickHouse users.xml as described in Section 2. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Command

Copy

$ cd /root -weight: 500;">git clone -b main https://github.com/SigNoz/signoz.-weight: 500;">git cd signoz/deploy/-weight: 500;">docker cd /root -weight: 500;">git clone -b main https://github.com/SigNoz/signoz.-weight: 500;">git cd signoz/deploy/-weight: 500;">docker cd /root -weight: 500;">git clone -b main https://github.com/SigNoz/signoz.-weight: 500;">git cd signoz/deploy/-weight: 500;">docker grep -o 'avx2' /proc/cpuinfo | head -1 # No output = no AVX2 support → apply the fix below grep -o 'avx2' /proc/cpuinfo | head -1 # No output = no AVX2 support → apply the fix below grep -o 'avx2' /proc/cpuinfo | head -1 # No output = no AVX2 support → apply the fix below <profiles> <default> <max_memory_usage>10000000000</max_memory_usage> <allow_simdjson>0</allow_simdjson> <!-- Fix for CPUs without AVX2 --> <load_balancing>random</load_balancing> </default> ... </profiles> <profiles> <default> <max_memory_usage>10000000000</max_memory_usage> <allow_simdjson>0</allow_simdjson> <!-- Fix for CPUs without AVX2 --> <load_balancing>random</load_balancing> </default> ... </profiles> <profiles> <default> <max_memory_usage>10000000000</max_memory_usage> <allow_simdjson>0</allow_simdjson> <!-- Fix for CPUs without AVX2 --> <load_balancing>random</load_balancing> </default> ... </profiles> Code: 173. DB::Exception: Couldn't allocate N bytes when parsing JSON Code: 173. DB::Exception: Couldn't allocate N bytes when parsing JSON Code: 173. DB::Exception: Couldn't allocate N bytes when parsing JSON <?xml version="1.0"?> <clickhouse> <asynchronous_metric_log -weight: 500;">remove="1"/> <trace_log -weight: 500;">remove="1"/> <metric_log -weight: 500;">remove="1"/> <query_views_log -weight: 500;">remove="1"/> <query_thread_log -weight: 500;">remove="1"/> <part_log -weight: 500;">remove="1"/> <processors_profile_log -weight: 500;">remove="1"/> <opentelemetry_span_log -weight: 500;">remove="1"/> </clickhouse> <?xml version="1.0"?> <clickhouse> <asynchronous_metric_log -weight: 500;">remove="1"/> <trace_log -weight: 500;">remove="1"/> <metric_log -weight: 500;">remove="1"/> <query_views_log -weight: 500;">remove="1"/> <query_thread_log -weight: 500;">remove="1"/> <part_log -weight: 500;">remove="1"/> <processors_profile_log -weight: 500;">remove="1"/> <opentelemetry_span_log -weight: 500;">remove="1"/> </clickhouse> <?xml version="1.0"?> <clickhouse> <asynchronous_metric_log -weight: 500;">remove="1"/> <trace_log -weight: 500;">remove="1"/> <metric_log -weight: 500;">remove="1"/> <query_views_log -weight: 500;">remove="1"/> <query_thread_log -weight: 500;">remove="1"/> <part_log -weight: 500;">remove="1"/> <processors_profile_log -weight: 500;">remove="1"/> <opentelemetry_span_log -weight: 500;">remove="1"/> </clickhouse> clickhouse: volumes: - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/ - ../common/clickhouse/cluster.xml:/etc/clickhouse-server/config.d/cluster.xml - ../common/clickhouse/ttl.xml:/etc/clickhouse-server/config.d/ttl.xml - ../common/clickhouse/z_log_disable.xml:/etc/clickhouse-server/config.d/z_log_disable.xml # ADD THIS LINE - clickhouse:/var/lib/clickhouse/ clickhouse: volumes: - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/ - ../common/clickhouse/cluster.xml:/etc/clickhouse-server/config.d/cluster.xml - ../common/clickhouse/ttl.xml:/etc/clickhouse-server/config.d/ttl.xml - ../common/clickhouse/z_log_disable.xml:/etc/clickhouse-server/config.d/z_log_disable.xml # ADD THIS LINE - clickhouse:/var/lib/clickhouse/ clickhouse: volumes: - ../common/clickhouse/config.xml:/etc/clickhouse-server/config.xml - ../common/clickhouse/users.xml:/etc/clickhouse-server/users.xml - ../common/clickhouse/custom-function.xml:/etc/clickhouse-server/custom-function.xml - ../common/clickhouse/user_scripts:/var/lib/clickhouse/user_scripts/ - ../common/clickhouse/cluster.xml:/etc/clickhouse-server/config.d/cluster.xml - ../common/clickhouse/ttl.xml:/etc/clickhouse-server/config.d/ttl.xml - ../common/clickhouse/z_log_disable.xml:/etc/clickhouse-server/config.d/z_log_disable.xml # ADD THIS LINE - clickhouse:/var/lib/clickhouse/ signoz: environment: - SIGNOZ_ALERTMANAGER_PROVIDER=signoz - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000 - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db # SMTP configuration - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM=your-email@gmail.com - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST=smtp.gmail.com:587 - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME=your-email@gmail.com - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD=your-16-char-app-password signoz: environment: - SIGNOZ_ALERTMANAGER_PROVIDER=signoz - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000 - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db # SMTP configuration - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM=your-email@gmail.com - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST=smtp.gmail.com:587 - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME=your-email@gmail.com - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD=your-16-char-app-password signoz: environment: - SIGNOZ_ALERTMANAGER_PROVIDER=signoz - SIGNOZ_TELEMETRYSTORE_CLICKHOUSE_DSN=tcp://clickhouse:9000 - SIGNOZ_SQLSTORE_SQLITE_PATH=/var/lib/signoz/signoz.db # SMTP configuration - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__FROM=your-email@gmail.com - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__SMARTHOST=smtp.gmail.com:587 - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__USERNAME=your-email@gmail.com - SIGNOZ_ALERTMANAGER_SIGNOZ_GLOBAL_SMTP__AUTH__PASSWORD=your-16-char-app-password cd /root/signoz/deploy/-weight: 500;">docker -weight: 500;">docker compose up -d cd /root/signoz/deploy/-weight: 500;">docker -weight: 500;">docker compose up -d cd /root/signoz/deploy/-weight: 500;">docker -weight: 500;">docker compose up -d -weight: 500;">docker compose ps -weight: 500;">docker compose ps -weight: 500;">docker compose ps -weight: 500;">docker exec signoz-clickhouse clickhouse-client --query " SELECT name FROM system.tables WHERE database='system' AND name LIKE '%log%' ORDER BY name" # Only query_log should remain -weight: 500;">docker exec signoz-clickhouse clickhouse-client --query " SELECT name FROM system.tables WHERE database='system' AND name LIKE '%log%' ORDER BY name" # Only query_log should remain -weight: 500;">docker exec signoz-clickhouse clickhouse-client --query " SELECT name FROM system.tables WHERE database='system' AND name LIKE '%log%' ORDER BY name" # Only query_log should remain mkdir -p /opt/otel-agent cat > /opt/otel-agent/config.yaml << 'EOF' receivers: hostmetrics: root_path: /hostfs collection_interval: 60s scrapers: cpu: {} disk: {} load: {} filesystem: exclude_mount_points: mount_points: [/snap/*, /boot/efi] match_type: regexp memory: {} network: {} paging: {} processes: {} processors: batch: resourcedetection: detectors: [env, system] system: hostname_sources: [os] resource/env: attributes: - key: deployment.environment value: production action: upsert exporters: otlp: endpoint: "<SIGNOZ_MONITOR_SERVER_IP>:4317" tls: insecure: true extensions: health_check: -weight: 500;">service: extensions: [health_check] pipelines: metrics: receivers: [hostmetrics] processors: [resourcedetection, resource/env, batch] exporters: [otlp] EOF mkdir -p /opt/otel-agent cat > /opt/otel-agent/config.yaml << 'EOF' receivers: hostmetrics: root_path: /hostfs collection_interval: 60s scrapers: cpu: {} disk: {} load: {} filesystem: exclude_mount_points: mount_points: [/snap/*, /boot/efi] match_type: regexp memory: {} network: {} paging: {} processes: {} processors: batch: resourcedetection: detectors: [env, system] system: hostname_sources: [os] resource/env: attributes: - key: deployment.environment value: production action: upsert exporters: otlp: endpoint: "<SIGNOZ_MONITOR_SERVER_IP>:4317" tls: insecure: true extensions: health_check: -weight: 500;">service: extensions: [health_check] pipelines: metrics: receivers: [hostmetrics] processors: [resourcedetection, resource/env, batch] exporters: [otlp] EOF mkdir -p /opt/otel-agent cat > /opt/otel-agent/config.yaml << 'EOF' receivers: hostmetrics: root_path: /hostfs collection_interval: 60s scrapers: cpu: {} disk: {} load: {} filesystem: exclude_mount_points: mount_points: [/snap/*, /boot/efi] match_type: regexp memory: {} network: {} paging: {} processes: {} processors: batch: resourcedetection: detectors: [env, system] system: hostname_sources: [os] resource/env: attributes: - key: deployment.environment value: production action: upsert exporters: otlp: endpoint: "<SIGNOZ_MONITOR_SERVER_IP>:4317" tls: insecure: true extensions: health_check: -weight: 500;">service: extensions: [health_check] pipelines: metrics: receivers: [hostmetrics] processors: [resourcedetection, resource/env, batch] exporters: [otlp] EOF -weight: 500;">docker run -d \ --name otel-agent \ ---weight: 500;">restart unless-stopped \ --hostname navio-main-server \ -v /opt/otel-agent/config.yaml:/etc/otelcol-contrib/config.yaml \ -v /:/hostfs:ro \ -e HOST_PROC=/hostfs/proc \ -e HOST_SYS=/hostfs/sys \ -e HOST_ETC=/hostfs/etc \ -e HOST_VAR=/hostfs/var \ -e HOST_RUN=/hostfs/run \ -e HOST_DEV=/hostfs/dev \ otel/opentelemetry-collector-contrib:latest -weight: 500;">docker run -d \ --name otel-agent \ ---weight: 500;">restart unless-stopped \ --hostname navio-main-server \ -v /opt/otel-agent/config.yaml:/etc/otelcol-contrib/config.yaml \ -v /:/hostfs:ro \ -e HOST_PROC=/hostfs/proc \ -e HOST_SYS=/hostfs/sys \ -e HOST_ETC=/hostfs/etc \ -e HOST_VAR=/hostfs/var \ -e HOST_RUN=/hostfs/run \ -e HOST_DEV=/hostfs/dev \ otel/opentelemetry-collector-contrib:latest -weight: 500;">docker run -d \ --name otel-agent \ ---weight: 500;">restart unless-stopped \ --hostname navio-main-server \ -v /opt/otel-agent/config.yaml:/etc/otelcol-contrib/config.yaml \ -v /:/hostfs:ro \ -e HOST_PROC=/hostfs/proc \ -e HOST_SYS=/hostfs/sys \ -e HOST_ETC=/hostfs/etc \ -e HOST_VAR=/hostfs/var \ -e HOST_RUN=/hostfs/run \ -e HOST_DEV=/hostfs/dev \ otel/opentelemetry-collector-contrib:latest Alert: Disk usage has exceeded {{$threshold}}% (current: {{$value}}%) Alert: Disk usage has exceeded {{$threshold}}% (current: {{$value}}%) Alert: Disk usage has exceeded {{$threshold}}% (current: {{$value}}%) # Check disk usage on monitor server df -h / # Verify ClickHouse system logs are not growing -weight: 500;">docker exec signoz-clickhouse clickhouse-client --query " SELECT table, formatReadableSize(sum(bytes_on_disk)) as size FROM system.parts WHERE database = 'system' GROUP BY table ORDER BY sum(bytes_on_disk) DESC" # Clean up unused Docker images on main server -weight: 500;">docker image prune -af # Check Docker disk usage -weight: 500;">docker system df # Check disk usage on monitor server df -h / # Verify ClickHouse system logs are not growing -weight: 500;">docker exec signoz-clickhouse clickhouse-client --query " SELECT table, formatReadableSize(sum(bytes_on_disk)) as size FROM system.parts WHERE database = 'system' GROUP BY table ORDER BY sum(bytes_on_disk) DESC" # Clean up unused Docker images on main server -weight: 500;">docker image prune -af # Check Docker disk usage -weight: 500;">docker system df # Check disk usage on monitor server df -h / # Verify ClickHouse system logs are not growing -weight: 500;">docker exec signoz-clickhouse clickhouse-client --query " SELECT table, formatReadableSize(sum(bytes_on_disk)) as size FROM system.parts WHERE database = 'system' GROUP BY table ORDER BY sum(bytes_on_disk) DESC" # Clean up unused Docker images on main server -weight: 500;">docker image prune -af # Check Docker disk usage -weight: 500;">docker system df # Enter clickhouse-client -weight: 500;">docker exec -it signoz-clickhouse clickhouse-client # Create force drop flag (run in a separate terminal) -weight: 500;">docker exec signoz-clickhouse bash -c \ "touch /var/lib/clickhouse/flags/force_drop_table && \ chmod 666 /var/lib/clickhouse/flags/force_drop_table" # Truncate all large log tables TRUNCATE TABLE system.trace_log; TRUNCATE TABLE system.metric_log; TRUNCATE TABLE system.query_log; TRUNCATE TABLE system.processors_profile_log; TRUNCATE TABLE system.part_log; TRUNCATE TABLE system.asynchronous_metric_log; TRUNCATE TABLE system.query_views_log; # Enter clickhouse-client -weight: 500;">docker exec -it signoz-clickhouse clickhouse-client # Create force drop flag (run in a separate terminal) -weight: 500;">docker exec signoz-clickhouse bash -c \ "touch /var/lib/clickhouse/flags/force_drop_table && \ chmod 666 /var/lib/clickhouse/flags/force_drop_table" # Truncate all large log tables TRUNCATE TABLE system.trace_log; TRUNCATE TABLE system.metric_log; TRUNCATE TABLE system.query_log; TRUNCATE TABLE system.processors_profile_log; TRUNCATE TABLE system.part_log; TRUNCATE TABLE system.asynchronous_metric_log; TRUNCATE TABLE system.query_views_log; # Enter clickhouse-client -weight: 500;">docker exec -it signoz-clickhouse clickhouse-client # Create force drop flag (run in a separate terminal) -weight: 500;">docker exec signoz-clickhouse bash -c \ "touch /var/lib/clickhouse/flags/force_drop_table && \ chmod 666 /var/lib/clickhouse/flags/force_drop_table" # Truncate all large log tables TRUNCATE TABLE system.trace_log; TRUNCATE TABLE system.metric_log; TRUNCATE TABLE system.query_log; TRUNCATE TABLE system.processors_profile_log; TRUNCATE TABLE system.part_log; TRUNCATE TABLE system.asynchronous_metric_log; TRUNCATE TABLE system.query_views_log; # Check AVX2 support grep -o 'avx2' /proc/cpuinfo | head -1 # No output = no AVX2 # Check AVX2 support grep -o 'avx2' /proc/cpuinfo | head -1 # No output = no AVX2 # Check AVX2 support grep -o 'avx2' /proc/cpuinfo | head -1 # No output = no AVX2 -weight: 500;">docker logs otel-agent 2>&1 | tail -20 # Check that host.name is detected correctly # Check that the endpoint IP is correct -weight: 500;">docker logs otel-agent 2>&1 | tail -20 # Check that host.name is detected correctly # Check that the endpoint IP is correct -weight: 500;">docker logs otel-agent 2>&1 | tail -20 # Check that host.name is detected correctly # Check that the endpoint IP is correct - OS: Ubuntu (snap -weight: 500;">docker) - Docker path: /var/snap/-weight: 500;">docker/common/var-lib--weight: 500;">docker/ - SigNoz path: /root/signoz/ - SigNoz version: v0.117.1 - ClickHouse version: 25.5.6 - CPU: No AVX2 support (important — affects configuration) - Infrastructure → Hosts tab (returns 500 error) - Any dashboard querying metrics with label filters - Metric: system.filesystem.usage - Filter: state = used AND host.name = <your-hostname> - Time aggregation: avg, Space aggregation: sum - Metric: system.filesystem.usage - Filter: host.name = <your-hostname> - Time aggregation: avg, Space aggregation: sum - Warning: above 80 - Critical: above 90 - Evaluation window: last 5 min

🏷️ Tags

toolsutilitiessecurity toolssignozsetupguidemonitorserver

More from Tools

Tools: Complete Guide to Anthropic Cybersecurity Skills: 754 playbooks de seguridad para agentes IA

Tools: Complete Guide to Anthropic Cybersecurity Skills: 754 playbooks de seguridad para agentes IA

2026-04-20 0
Tools: Docker for Beginners: Containerize Your App in 30 Minutes

Tools: Docker for Beginners: Containerize Your App in 30 Minutes

2026-04-20 0
Tools: Complete Guide to Docker Compose Explained: Multi-Container Stacks (2026)

Tools: Complete Guide to Docker Compose Explained: Multi-Container Stacks (2026)

2026-04-20 0
Tools: Essential Guide: Cheapest VPS $5/Month: What You Really Get

Tools: Essential Guide: Cheapest VPS $5/Month: What You Really Get

2026-04-20 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views