Este es un articulo en donde se pretende enviar todos los logs a rsyslog central donde se almacenen los logs y se creen gráficas con kibana. El flujo de datos es el siguiente:

Nodos cliente –> rsyslog central –> rsyslog central remoto –> logstash –> elasticsearch –> Kibana.

apt install rsyslog
systemctl  restart rsyslog
man rsyslogd
man rsyslog.conf
semanage -a -t syslogd_port_t -p udp 514
semanage -a -t syslogd_port_t -p tcp 514 
ufw allow 514/udp
ufw allow 514/tcp

/etc/rsyslog.conf 
/etc/default/rsyslog

cat /etc/rsyslog.conf 
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")
 
# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")
 
cat /etc/rsyslog.d/40-server.conf 
$template RemoteLogs,"/var/log/rsyslog/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?RemoteLogs
#&  ~  #Put this to stop logging
 
 
cat /etc/rsyslog.d/60-cliente.conf 
*.* @@192.168.0.181:514

apt install mariadb-client mariadb-server

apt install rsyslog gnutls-bin rsyslog-gnutls rsyslog-mysql

Crear los certificados

mkdir -p /etc/certs/gnutls
cd /etc/certs/gnutls
 
#Creando la entidad certificadora
certtool --generate-privkey --outfile ca-key.pem
certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem
 
#Creando el certificado cliente
certtool --generate-privkey --outfile key.pem
certtool --generate-request --load-privkey key.pem --outfile request.pem
certtool --generate-certificate --load-request request.pem --outfile cert.pem  --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem

Editar el archivo /etc/rsyslog.conf con nano Y Agregar en la seccion global este bloque

# make gtls driver the default and set certificate files
global(
DefaultNetstreamDriver="gtls"
DefaultNetstreamDriverCAFile="/etc/certs/gnutls/ca.pem"
DefaultNetstreamDriverCertFile="/etc/certs/gnutls/cert.pem"
DefaultNetstreamDriverKeyFile="/etc/certs/gnutls/key.pem"
)
 
# load TCP listener
module(
load="imtcp"
StreamDriver.Name="gtls"
StreamDriver.Mode="1"
StreamDriver.Authmode="anon"
)
 
# start up listener at port 6514
input(
type="imtcp"
port="6514"
)

Reiniciar el Rsyslog y ver el resultado

systemctl restart rsyslog
tail /var/log/syslog

apt install rsyslog rsyslog-gnutls

Copiar el archivo /etc/certs/gnutls/ca.pem al cliente.

Editar el archivo /etc/rsyslog.conf con nano Y Agregar en la seccion global este bloque

# make gtls driver the default and set certificate files
# certificate files - just CA for a client
global(DefaultNetstreamDriverCAFile="/etc/certs/gnutls/ca.pem")
 
# set up the action for all messages
action(type="omfwd" Target="LaIPdelServidorLogs" protocol="tcp" port="6514"
       StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="anon")

Reiniciar el Rsyslog y ver el resultado

systemctl restart rsyslog
tail /var/log/syslog

Luego probamos que se envian los logs con

logger esta es una prueba

Y luego ver en el servidor de Logs esa linea.

apt install certbot python3-certbot-nginx nginx-light php-fpm php-gd php-mysql loganalyzer

Editar el archivo /etc/loganalyzer/config.php con nano Modificar los siguiente valores de la base de datos con los datos que estan en el archivo /etc/rsyslog.d/mysql.conf

        $CFG['Sources']['Source1']['ID'] = "Source1";
        $CFG['Sources']['Source1']['Name'] = "Mariadb";
        $CFG['Sources']['Source1']['Description'] = "Base de datos central";
        $CFG['Sources']['Source1']['SourceType'] = SOURCE_DB;
        $CFG['Sources']['Source1']['MsgParserList'] = "";
        $CFG['Sources']['Source1']['DBTableType'] = "winsyslog";
        $CFG['Sources']['Source1']['DBType'] = DB_MYSQL;
        $CFG['Sources']['Source1']['DBServer'] = "localhost";
        $CFG['Sources']['Source1']['DBName'] = "Syslog";
        $CFG['Sources']['Source1']['DBUser'] = "rsyslog";
        $CFG['Sources']['Source1']['DBPassword'] = "LaSuperClabe";
        $CFG['Sources']['Source1']['DBTableName'] = "SystemEvents";

Instalación de Java (Requisitos de ELK)

apt-get install default-jre
 
java version

Configuración de Repositorio de ELK

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
cat /etc/apt/sources.list.d/elastic-7.x.list
apt-get update

Instalando ELK

apt-get install elasticsearch logstash kibana
 
systemctl status elasticsearch
systemctl status logstash
systemctl status kibana
 
systemctl enable elasticsearch
systemctl enable logstash
systemctl enable kibana
 
 
systemctl start elasticsearch
systemctl start logstash
systemctl start kibana

Verificando puertos de ELK

#elastisearch por
lsof -i -P -n | grep LISTEN | grep 9200
 
#logstash port
lsof -i -P -n | grep LISTEN | grep 9600
 
#Kibana port
lsof -i -P -n | grep LISTEN | grep 5601

Enrutando de Logstash a Elasticsearch

input {                                                                                      
  udp {                                                                                      
    host => "127.0.0.1"                                                                      
    port => 10514                                                                            
    codec => "json"                                                                          
    type => "rsyslog"                                                                        
  }                                                                                          
}                                                                                            
 
 
# The Filter pipeline stays empty here, no formatting is done.                                                                                           filter { }                                                                                   
 
 
# Every single log will be forwarded to ElasticSearch. If you are using another port, you should specify it here.                                                                                             
output {                                                                                     
  if [type] == "rsyslog" {                                                                   
    elasticsearch {                                                                          
      hosts => [ "127.0.0.1:9200" ]                                                          
    }                                                                                        
  }                                                                                          
}                                                                                            

Reiniciando logstash y verificando servicio

systemctl restart logstash
netstat -na | grep 10514

Enrutando de Rsyslog a Logstash

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}
 
# This line sends all lines to defined IP address at port 10514
# using the json-template format.
 
*.*                         @127.0.0.1:10514;json-template

Reiniciando Rsyslog y verificando que recibimos datos en Logstash

systemctl restart rsyslog
curl -XGET 'http://localhost:9200/logstash-*/_search?q=*&pretty'

Luego solo falta crear el dashboard, panel y conexion a los datos en Kibana

Agregar esta configuración de nginx para poner loganalyzer y kibana en subfolders

      location ^~ /loganalizer/ {
           alias /usr/share/loganalyzer/;
           index index.php;
           location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                fastcgi_param  SCRIPT_FILENAME $request_filename;
           }
 
        }
 
        location ~ /kibana {
            proxy_pass http://localhost:5601;
        }

Para Debian bullseye 11

Instalación de requisitos

apt update
apt upgrade
sudo apt install apt-transport-https openjdk-11-jre-headless uuid-runtime pwgen dirmngr gnupg wget git certbot

Instalación de mongodb

wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/4.2 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
 
apt-get update
sudo apt-get install -y mongodb-org
 
systemctl daemon-reload
systemctl enable mongod.service
systemctl restart mongod.service
systemctl --type=service --state=active | grep mongod

Instalación de Elasticsearch

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/oss-7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update && sudo apt install elasticsearch-oss
apt update && sudo apt install elasticsearch-oss

Configuración de Elasticsearch

cat /etc/elasticsearch/elasticsearch.yml
tee -a /etc/elasticsearch/elasticsearch.yml > /dev/null << EOT
cluster.name: graylog
action.auto_create_index: false
EOT
 
cat  /etc/elasticsearch/elasticsearch.yml
 
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl restart elasticsearch.service
systemctl restart elasticsearch.service

Instalación de graylog

wget https://packages.graylog2.org/repo/packages/graylog-4.2-repository_latest.deb
dpkg -i graylog-4.2-repository_latest.deb
apt-get update && sudo apt-get install graylog-server graylog-enterprise-plugins graylog-integrations-plugins graylog-enterprise-integrations-plugins
 
systemctl enable graylog-server.service
systemctl start graylog-server.service
 
apt remove graylog-enterprise-plugins graylog-enterprise-integrations-plugins

Creamos la carpeta donde vamos a guardar los certificados

mkdir certificados
cd certificados

Creamos la llave para nuestro CA

certtool --generate-privkey --outfile ca-key.pem

Creamos el certificado CA con las siguientes respuestas:

Generating a self signed certificate...
Please enter the details of the certificate's distinguished name. Just press enter to ignore a field.
Common name: auth.ejemplo.com
UID: 
Organizational unit name: authority 
Organization name: ejemplo
Locality name: San Salvador
State or province name: San Salvador
Country name (2 chars): SV
Enter the subject's domain component (DC): auth.ejemplo.com
Enter an additional domain component (DC): 
This field should not be used in new certificates.
E-mail: 
Enter the certificate's serial number in decimal (123) or hex (0xabcd)
(default is 0x3efb2af4747406ca4a4d6e8b264f651c0b41fd4e)
value: 


Activation/Expiration time.
The certificate will expire in (days): 3651


Extensions.
Does the certificate belong to an authority? (y/N): y
Path length constraint (decimal, -1 for no constraint): 
Is this a TLS web client certificate? (y/N): 
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this a TLS web server certificate? (y/N): 
Enter a dnsName of the subject of the certificate: auth.ejemplo.com
Enter an additional dnsName of the subject of the certificate: 
Enter a URI of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Enter the e-mail of the subject of the certificate: 
Will the certificate be used for signing (required for TLS)? (Y/n): y
Will the certificate be used for data encryption? (y/N): 
Will the certificate be used to sign OCSP requests? (y/N): 
Will the certificate be used to sign code? (y/N): 
Will the certificate be used for time stamping? (y/N): 
Will the certificate be used for email protection? (y/N): 
Will the certificate be used to sign other certificates? (Y/n): y
Will the certificate be used to sign CRLs? (y/N): 
Enter the URI of the CRL distribution point: 

certtool --generate-self-signed --load-privkey ca-key.pem --outfile ca.pem

Cambiamos permisos a la llave Importante no copiar este archivo que es la que nos da la seguridad.

chmod 400 ca-key.pem
ls -alh

Creamos la llave para los clientes

certtool --generate-privkey --outfile key.pem --bits 2048

Creamos la solicitud de certificado usando el CA para los clientes con estas respuestas:

** Note: You may use '--sec-param Medium' instead of '--bits 2048'
Generating a 2048 bit RSA private key...
root@logs:~/certificados/new# certtool --generate-request --load-privkey key.pem --outfile request.pem
Generating a PKCS #10 certificate request...
Common name: client.ejemplo.com
Organizational unit name: client
Organization name: ejemplo
Locality name: San Salvador
State or province name: San Salvador
Country name (2 chars): SV
Enter the subject's domain component (DC): client.ejemplo.com
Enter an additional domain component (DC): 
UID: 
Enter a dnsName of the subject of the certificate: client.ejemplo.com
Enter an additional dnsName of the subject of the certificate: 
Enter a URI of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Enter the e-mail of the subject of the certificate: 
Enter a challenge password: 
Does the certificate belong to an authority? (y/N): n
Will the certificate be used for signing (DHE ciphersuites)? (Y/n): 
Will the certificate be used for encryption (RSA ciphersuites)? (Y/n): 
Will the certificate be used to sign code? (y/N): 
Will the certificate be used for time stamping? (y/N): 
Will the certificate be used for email protection? (y/N): 
Will the certificate be used for IPsec IKE operations? (y/N): 
Will the certificate be used to sign OCSP requests? (y/N): 
Is this a TLS web client certificate? (y/N): y
Is this a TLS web server certificate? (y/N): y

certtool --generate-request --load-privkey key.pem --outfile request.pem
ls -alh

Creamos el certificado para los clientes con estas respuestas:

Generating a signed certificate...
Enter the certificate's serial number in decimal (123) or hex (0xabcd)
(default is 0x29a9678b6734082b227281cc331037bfb1cac595)
value: 


Activation/Expiration time.
The certificate will expire in (days): 3650


Extensions.
Do you want to honour all the extensions from the request? (y/N): 
Does the certificate belong to an authority? (y/N): n
Is this a TLS web client certificate? (y/N): y
Will the certificate be used for IPsec IKE operations? (y/N): 
Is this a TLS web server certificate? (y/N): y
Enter a dnsName of the subject of the certificate: client.ejemplo.com
Enter an additional dnsName of the subject of the certificate: 
Enter a URI of the subject of the certificate: 
Enter the IP address of the subject of the certificate: 
Will the certificate be used for signing (DHE ciphersuites)? (Y/n): 
Will the certificate be used for encryption (RSA ciphersuites)? (Y/n): 
Will the certificate be used for data encryption? (y/N): 
Will the certificate be used to sign OCSP requests? (y/N): 
Will the certificate be used to sign code? (y/N): 
Will the certificate be used for time stamping? (y/N): 
Will the certificate be used for email protection? (y/N): 

certtool --generate-certificate --load-request request.pem --outfile cert.pem --load-ca-certificate ca.pem --load-ca-privkey ca-key.pem

Ahora borramos la solicitud de certificado

rm -f request.pem

Creamos la carpeta para los clientes

mkdir certificados-cliente
cp ca.pem  cert.pem  key.pem certificados-cliente/

Estas es la carpeta que vamos a copiar a nuestros clientes

Configuración de Graylog

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
 
nano /etc/graylog/server/server.conf
 
cat /etc/graylog/server/server.conf |grep -P ""^[^#]  
is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = unadelascadenasmasgrandes
root_password_sha2 = lacadenaquesacamosenelcomandoanterior
bin_dir = /usr/share/graylog-server/bin
data_dir = /var/lib/graylog-server
plugin_dir = /usr/share/graylog-server/plugin
http_publish_uri = http://localhost:9000/
http_external_uri =https://logs.ejemplo.com/graylog/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
proxied_requests_thread_pool_size = 32

Instalamos el soporte para certificados TLS

apt install -y rsyslog-gnutls

Copiamos los certificados

cp -rp ../certificados /etc/

Agregamos la configuración el rsyslog para que envie los logs al servidor central

cat  >> /etc/rsyslog.conf << EOL
 
#Para el servidor de logs
\$DefaultNetstreamDriver gtls
#$DefaultNetstreamDriver ossl
 
\$DefaultNetstreamDriverCAFile   /etc/certificados/new/ca.pem
\$DefaultNetstreamDriverCertFile /etc/certificados/new/cert.pem
\$DefaultNetstreamDriverKeyFile  /etc/certificados/new/key.pem
 
\$ActionSendStreamDriverAuthMode x509/name
\$ActionSendStreamDriverPermittedPeer auth.ejemplo.com
\$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
 
# Increase the amount of open files rsyslog is allowed, which includes open tcp sockets
# This is important if there are many clients.
# http://www.rsyslog.com/doc/rsconf1_maxopenfiles.html
#$MaxOpenFiles 2048
 
*.*@@logs.ejemplo.com:8514;RSYSLOG_SyslogProtocol23Format
EOL

Reiniciamos y verificamos que no haya problemas

systemctl restart rsyslog
 
tail -n 50 /var/log/syslog

• https://www.tecmint.com/install-rsyslog-centralized-logging-in-centos-ubuntu/
• https://www.rsyslog.com/doc/v8-stable/tutorials/database.html
• https://www.howtoforge.com/tutorial/rsyslog-centralized-log-server-in-debian-9/
• https://wiki.debian.org/Rsyslog
• https://www.caktusgroup.com/blog/2019/06/20/how-to-set-up-centralized-log-server-rsyslog/
• https://www.thegeekdiary.com/how-to-configure-rsyslog-server-to-accept-logs-via-ssl-tls/
• https://rsyslog.readthedocs.io/en/latest/tutorials/tls.html
• https://www.rsyslog.com/doc/v8-stable/tutorials/tls.html
• https://www.rsyslog.com/files/temp/doc-indent/configuration/modules/omfwd.html
• https://devconnected.com/the-definitive-guide-to-centralized-logging-with-syslog-on-linux/
• https://devconnected.com/monitoring-linux-logs-with-kibana-and-rsyslog/
• https://www.graylog.org/
• https://docs.graylog.org/v1/docs/debian
• https://computingforgeeks.com/install-graylog-on-debian-with-lets-encrypt-ssl/
• https://docs.graylog.org/v1/docs/syslog