Today, I am very excited to show you guys how to install Netdata and have fun with it! For those who don't know what Netdata is, it is a real-time health monitoring and performance troubleshooting tool for your systems and applications.
To find out more about Netdata, please check out their well-written documentation at https://docs.netdata.cloud/.
If you are a rookie administrator like me, you probably don't know too much about how your system is running. What you might care is whether the system is up and running, and if the application stops running or website is down, your possibly best solution might be performing a system reboot or restart the application. Ironically, you might only find out about it after it is stopped working, and most likely you were told by the users or you have received some kind of alerts after the incident.
You might know some basic Linux command line prompts to check on the system status, for example, top and ps -aux to view the processes, df -h to view the storage, free -m to check on memory usage, tail -f to view the logs and so on. Let's be honest here, you don't know much about them and you probably just look them up and try your luck here. It is such a pain whenever the system is having some issues and you don't even know what exactly went wrong and where to look into in the first place. Anyway, I am so glad to find out Netdata - it is free, open source project with easy installation, active development, strong community, and well-written documentations.
Without further ado, let's get into it!
1. SSH into your instance and run one line installation script
$ bash <(curl -Ss https://my-netdata.io/kickstart.sh)
You can find out the installation guide here.
NOTE: If you are trying the script on your Amazon EC2 Instance with older image version (Amazon Linux AMI), the suggested one line installation script might not work. You will probably see the build is failed.
You can try to run the script kickstart-static64.sh instead:
$ bash <(curl -Ss https://my-netdata.io/kickstart-static64.sh)
Not sure why but it works in my case though. If you can see the screenshot below, then congratulations to you! You have installed Netdata successfully!
2. Check if Netdata is installed successfully
You can visit the link http://your-server-ip-address:19999 to find out the Netdata dashboard.
For example, if your server IP is 33.53.23.23, then you can type http://33.53.23.23:19999 in the browser URL bar, you will see something like this:
We will talk about how to access the link via domain name in a bit.
19999 in your instance's security group. For now, we can add the inbound rules for port 19999 to test. We will not need to open up this port in section later.
3. Edit the config file
Now, let's check out the Netdata config file. The config file is located at /opt/netdata/netdata-configs/netdata.conf.
By default, you are not able to access the netdata.conf at http://<netdata_ip>/netdata.conf. You might see the text saying "You are not allowed to access this resource." simply because it is protected and unavailable to public. We can configure the value in the config file so that the file can be assessible through URL.
Anyway, let's dive into the netdata.conf file:
$ cd /opt/netdata/netdata-configs
$ sudo ./edit-config netdata.conf
Running the command line ./edit-config will open up the file in editor just like you open it with any of your favorite editor such as vim, except using ./edit-config to edit is claimed to be safer and it will automatically grab the default config file so that you can better keep track of what you have been editing. So after you open up the file and done modifying, you can save and exit. Here's what you will see after the command line:
[global]
# glibc malloc arena max for plugins = 1
# hostname = ip-10-2-200-213
# history = 3996
# update every = 1
# config directory = /opt/netdata/etc/netdata
# stock config directory = /opt/netdata/usr/lib/netdata/conf.d
# log directory = /opt/netdata/var/log/netdata
# web files directory = /opt/netdata/usr/share/netdata/web
# cache directory = /opt/netdata/var/cache/netdata
...
(too long and I am just going to omit the rest...)
Say, I want to change the hostname, I can type this command line and modify the file by uncommenting the hostname line and set the hostname I want like this:
[global]
# glibc malloc arena max for plugins = 1
hostname = test-home
# history = 3996
...
Once you have finished editing the file, you can run this command sudo service netdata restart to restart Netdata. Of course, feel free to change any of the config value as needed.
/opt/netdata/netdata-configs when you are running command sudo ./edit-config netdata.conf with sudo. Also, in order for the modified config file to come info effect, you need to restart Netdata.
Anyway, you will be able to see that your hostname is changed to the value you set previously in the config file:
4. Set the receiver for alarm alerts
Next, we will edit this config file health_alarm_notify.conf. Make sure we are still in /opt/netdata/netdata-configs folder. We can edit this file by command sudo ./edit-config health_alarm_notify.conf. Search for DEFAULT_RECIPIENT_EMAIL and set your own email address.
...
# enable/disable sending emails
SEND_EMAIL="YES"
# if a role recipient is not configured, an email will be send to:
DEFAULT_RECIPIENT_EMAIL="john@example.com"
# to receive only critical alarms, set it to "root|critical"
...
If you want to enable Slack, you can register a Slack Webhook. Once you have completed setting up the webhook, grab the Slack Webhook URL and set it in the config file.
...
# You need only one for all your netdata servers (or you can have one for each of your netdata).
# Without the app and a webhook, netdata cannot send slack notifications.
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/XXXXXXXXXXXX"
...
To test it out, you can login as netdata user via bash and run the script.
$ sudo su -s /bin/bash netdata
$ /opt/netdata/usr/libexec/netdata/plugins.d/alarm-notify.sh test
You will be receiving test email and slack notification (if you have setup both of them).
5. Apache/Nginx monitoring with Netdata
To allow Apache monitoring with Netdata, you need to make sure that you have mod_status setup. To learn more about mod_status, please check out the official Apache website.
sudo yum update httpd to update Apache if needed.
Long story short, what you need to do is to modify the httpd.conf and add the following lines:
ExtendedStatus on
<Location /server-status>
SetHandler server-status
order deny,allow
deny from all
allow from 127.0.0.1
</Location>
The httpd.conf file is usually located at /etc/httpd/conf. Make sure you restart the service to apply the update. Just for your reference, you can run sudo service httpd restart to restart Apache.
Once we have done that, we need to go back to netdata-configs folder and run the command sudo ./edit-config python.d/apache.conf (not necessarily, but in case you need to update the config file).
Make sure the URL http://localhost/server-status is accessible by Netdata application though.
server-status in your .htaccess file (the .htaccess file is usually located in your Wordpress root directory /var/www/html). You can add this RewriteCond %{REQUEST_URI} !=/server-status right before the RewriteRule . /index.php [L] to bypass the redirection.
Once you have done all this, you will see a new section called Apache local available for you.
Another cool thing that you will see is the web log apache section. This section will be available once Netdata app can access to access_log file in /var/log/httpd folder. To provide the permission in the simplest way is to change the folder's owner/group as well as the access_log file.
$ sudo chown netdata:netdata /var/log/httpd
$ sudo chown netdata:netdata /var/log/httpd/access_log
You probably don't need to change or modify anything since the config file has already taken care of almost all the possibilities. But if you do, feel free to run ./edit-config python.d/web_log.conf to edit.
Likewise for Nginx setup, first you need to make sure you have stub_status directive set by running the following command:
$ nginx -V 2>&1 | grep -o with-http_stub_status_module
If the commands return nothing, that means such module is not enabled yet. You might want to check out the Nginx documentation.
Once you confirm the module is enabled, you can modify your /etc/nginx/nginx.conf by adding these lines:
server {
listen 127.0.0.1:80;
server_name 127.0.0.1;
location /stub_status {
stub_status;
}
}
Make sure the URL http://localhost/stub_status is accessible by Netdata application though.
6. Refine the Apache log
Since the web log apache reads from the access_log file, there might be quite a lot of unhelpful information that we want to filter out. In the /etc/httpd/conf/httpd.conf file, find the line says LogFormat and add these following lines:
...
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
SetEnvIf Request_URI "^/server-status" dontlog
SetEnvIf User-Agent "ELB-HealthChecker/2.0" dontlog
SetEnvIf Remote_Addr "127.0.0.1" dontlog
SetEnvIf Remote_Addr "::1" dontlog
SetEnvIf Referer "netdata" dontlog
CustomLog "logs/access_log" combined env=!dontlog
...
Let me explain it.
-
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedis the default one came with installation. You don't need to change unless if you want to. You can find out more about each parameter in the Apache website. -
SetEnvIf Request_URI "^/server-status" dontlogexcludes the request url starts with/server-status, ex:http://localhost/server-status. -
SetEnvIf User-Agent "ELB-HealthChecker/2.0" dontlogexcludes the request with the specific agent name. -
SetEnvIf Remote_Addr "127.0.0.1" dontlogandSetEnvIf Remote_Addr "::1" dontlogexcludes the request coming from localhost (loopback address foripv4andipv6). -
SetEnvIf Referer "netdata" dontlogexcludes referer contains "netdata" keyword (depends on how you setup the domain though, I have set the subdomain URL with "netdata" in it and I want to exclude that) -
CustomLog "logs/access_log" combined env=!dontlogmakes sure thoseenvthat we set is not logged.
That's it! Restart your httpd to see if it comes into effect. Feel free to play around with it.
7. Set up domain and strip the port
Now, we want to visit Netdata dashboard without manually typing the IP address, how can we do that? Well, it's very straightforward. You can just add a A Record and set the server IP address as the destination will do.
Previously, you might need to type like this: http://33.53.23.23:19999.
Now, you can just insert the domain name with the port: http://netdata.domain.com:19999.
You still don't feel pretty about the port part and you want to get rid of it.
In Apache setup, we can achieve that by setting a proxy behind it. We don't need to install another nginx engine to achieve this, all we need to do is to modify our httpd.conf file and add these lines:
<VirtualHost *:80>
ServerName netdata.domain.com
ServerAlias netdata.domain.com
ProxyPreserveHost On
ProxyPass / http://localhost:19999/
ProxyPassReverse / http://localhost:19999/
</VirtualHost>
<VirtualHost *:80>
ServerName www.domain.com
ServerAlias www.domain.com
</VirtualHost>
We only want the domain netdata.domain.com to be routed to Netadata application, hence you need to set up at least 2 virtual hosts, one with ServerName netdata.domain.come and the other one with different ServerName of yours. If you don't do so, all of the requests will be proxied to Netdata application by default.
Make sure the proxy module is included though. It is usually included in the config file by default actually. Restart your httpd service and check if that works.
In Nginx setup, the nginx.conf can be adding the following lines:
upstream netdata-backend {
# the Netdata server
server 127.0.0.1:19999;
keepalive 64;
}
server {
listen 80;
server_name netdata.example.com;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://netdata-backend;
proxy_http_version 1.1;
proxy_pass_request_headers on;
proxy_set_header Connection "keep-alive";
proxy_store off;
}
}
8. Authentication for your Netdata
You might question that there is no sort of protection to visit the dashboard, anyone with the IP address or the domain URL will be able to access to the dashboard. Well, good thing is that they don't have the permission to edit anything since it is read only. You still don't want them to find out how your server is doing, then what you can do is to setup a simple basic authentication before they can access to the dashboard.
The simplest way to protect your dashboard is to access it by IP with port number and use Security Group to restrict on port 19999 access to your own IP address or company's IP address, but this is not what I am talking about. I mean sure, you can go down that path but I am just going to get a little fancier.
If your web server application is Nginx, you can visit this link for more information.
For Apache web server, first we need to create a user:
$ sudo htpasswd -c /etc/httpd/.htpasswd test
New password:
Re-type new password:
Adding password for user test
-c option is to create .htpasswd file. The next time you add new user, you might want to leave out -c otherwise your previous .htpasswd might be replaced.
Next, you just need to add the following <Location "/">...</Location> into virtual host in httpd.conf:
# the subdomain that needs to proxy to port 19999
<VirtualHost *:80>
ServerName netdata.domain.com
ServerAlias netdata.domain.com
<Location "/">
AuthType Basic
AuthName "Restricted Content"
AuthUserFile /etc/httpd/.htpasswd
Require valid-user
</Location>
ProxyPreserveHost On
ProxyPass / http://localhost:19999/
ProxyPassReverse / http://localhost:19999/
</VirtualHost>
# allow netdata to make localhost request
<VirtualHost 127.0.0.1:80>
ServerName netdata.domain.com
ServerAlias netdata.domain.com
</VirtualHost>
# other subdomain
<VirtualHost *:80>
ServerName www.domain.com
ServerAlias www.domain.com
</VirtualHost>
Once you have done that, you can restart httpd service and check if it works.
For Nginx web server, we can create new user by using openssl:
sudo sh -c "echo -n 'issac:' >> /etc/nginx/.htpasswd"
sudo sh -c "openssl passwd -apr1 >> /etc/nginx/.htpasswd"
Likewise, you can modify the nginx.conf located in /etx/nginx like this:
upstream netdata-backend {
# the Netdata server
server 127.0.0.1:19999;
keepalive 64;
}
server {
listen 80;
server_name netdata.example.com;
auth_basic "Restricted Content";
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://netdata-backend;
proxy_http_version 1.1;
proxy_pass_request_headers on;
proxy_set_header Connection "keep-alive";
proxy_store off;
}
}
You might ask, it is still not secure though. People might steal your login information since the URL is not protected by HTTPS. If you want to setup SSL for protection, you can check out this link to get a gist of it. I am using Elastic Load Balancer (ELB) which already handles the HTTPS for me.
9. Manage all your Netdatas in one place
Finally, this is the last step of this tutorial. If you want to manage all of your Netdata dashboards in one place, you can make use of Nodes. Nodes is currently in Beta, but you can give it a try and see how it works.
Microsoft Edge.
You can click the Sign In button on the top-right corner. You can choose to login via email address, or OAuth via Github or Google.
Once you login, you will be able to find all of your nodes and view them altogether:
You can also view the nodes in console:
If you are having issue displaying other nodes in the console, probably it is because of the authentication that you set earlier which prohibits all the access to be authenticated.
<Location "/">
# if referrer contains "netdata" and "example.com", no need to authenticate
<If " %{HTTP_HOST} =~ /example.com/ && %{HTTP_REFERER} =~ /netdata/ && %{HTTP_REFERER} =~ m#example.com/console.html# ">
</If>
<Else>
AuthType Basic
AuthName "Restricted Content"
AuthUserFile /etc/httpd/.htpasswd
Require valid-user
</Else>
</Location>
In the example above, I excluded those requests with referer of netdata keyword so that the console can work properly. Visit Apache website to find out more on how to use expression in Apache.
In Nginx, you can modify your nginx.conf:
server {
listen 80;
server_name netdata.example.com;
set $auth_basic "Restricted Content";
if ( $http_referer ~ netdata ){
set $access_code N;
}
if ( $http_referer ~ example\.com\/console.html ){
set $access_code "${access_code}M";
}
if ( $http_host ~ example\.com ){
set $access_code "${access_code}H";
}
if ( $access_code = NMH ){
set $auth_basic off;
}
auth_basic $auth_basic;
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://netdata-backend;
proxy_http_version 1.1;
proxy_pass_request_headers on;
proxy_set_header Connection "keep-alive";
proxy_store off;
}
}
To be honest, it is not as intuitive as the Apache's solution but at least it is doing what it is supposed to do.
That's all about it! Pretty amazing, right? Of course, please check out their official documentation and github too!
Well, I hope this post can be useful for you too. See you next time!
Post was published on , last updated on .
Like the content? Support the author by paypal.me!
