NavigationContentFooter
Jump toSuggest an edit
Was this page helpful?

Setting up the Elastic Stack (formerly ELK Stack)

Reviewed on 06 March 2025Published on 10 June 2015
  • ELK-stack
  • ELK
  • elastic-stack
  • elasticsearch
  • logstash
  • kibana

The Elastic Stack, formerly known as the ELK Stack, is a powerful suite of open-source tools designed for real-time data search, analysis, and visualization. It offers comprehensive capabilities for collecting, processing, and visualizing large volumes of data. Its components are:

  • Elasticsearch: A distributed, RESTful search and analytics engine based on the Lucene library.
  • Logstash: A flexible data collection, processing, and enrichment pipeline.
  • Kibana: A visualization and exploration tool for analyzing and visualizing data stored in Elasticsearch.
  • Beats: Lightweight data shippers for ingesting data into Elasticsearch or Logstash.

Before you startLink to this anchor

To complete the actions presented below, you must have:

  • A Scaleway account logged into the console
  • Owner status or IAM permissions allowing you to perform actions in the intended Organization
  • An SSH key
  • An Instance or an Elastic Metal server with at least 4 GB of RAM

Install ElasticsearchLink to this anchor

  1. Download and install the Elasticsearch signing key:

    curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-archive-keyring.gpg
  2. Add the Elasticsearch repository:

    echo "deb [signed-by=/usr/share/keyrings/elasticsearch-archive-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-8.x.list
  3. Update the apt package repositories:

    apt update
  4. Install Elasticsearch:

    apt install elasticsearch
  5. Start and enable the Elasticsearch service:

    systemctl start elasticsearch
    systemctl enable elasticsearch
  6. Configure Elasticsearch for production: Modify the elasticsearch.yml file to optimize Elasticsearch for production use:

    nano /etc/elasticsearch/elasticsearch.yml

    Add the following:

    cluster.name: "my-cluster"
    node.name: "node-1"
    network.host: 0.0.0.0
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/keystore.p12
    xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/truststore.p12
    Note

    Make sure you have SSL certificates set up for secure communication.

Install and configure LogstashLink to this anchor

  1. Install Logstash using the same repository added for Elasticsearch:

    apt install logstash
  2. Create and modify configuration files for Logstash: The configuration files for Logstash are typically located in /etc/logstash/conf.d/. You can create pipelines to manage your data processing.

  3. Start and enable the Logstash service:

    systemctl start logstash
    systemctl enable logstash

Install and configure KibanaLink to this anchor

  1. Install Kibana:

    apt install kibana
  2. Start and enable the Kibana service:

    systemctl start kibana
    systemctl enable kibana
  3. Configure Kibana for remote access: By default, Kibana is accessible on http://localhost:5601. To make Kibana accessible remotely, edit the Kibana configuration file:

    nano /etc/kibana/kibana.yml

    Change the server host to:

    server.host: "0.0.0.0"
  4. Secure Kibana: Ensure Kibana uses SSL to encrypt communications by adding SSL certificates in the kibana.yml file:

    server.ssl.enabled: true
    server.ssl.certificate: /etc/kibana/certs/kibana.crt
    server.ssl.key: /etc/kibana/certs/kibana.key
    elasticsearch.ssl.certificate: /etc/kibana/certs/kibana.crt
    elasticsearch.ssl.key: /etc/kibana/certs/kibana.key

Install and configure FilebeatLink to this anchor

  1. Install Filebeat:

    apt install filebeat
  2. Configure Filebeat to ship logs to Elasticsearch: Edit the Filebeat configuration file to point to your Elasticsearch instance:

    nano /etc/filebeat/filebeat.yml

    Set the output to Elasticsearch:

    output.elasticsearch:
    hosts: ["http://localhost:9200"]

    Alternatively, configure Filebeat to send logs to Logstash:

    output.logstash:
    hosts: ["localhost:5044"]
  3. Start and enable the Filebeat service:

    systemctl enable filebeat
    systemctl start filebeat

Secure the Elastic StackLink to this anchor

Securing your Elastic Stack is essential, especially if exposed to the internet. Following are some recommendations:

  • Enable built-in security features (as shown above in Elasticsearch and Kibana setup).

  • Use a firewall: You can use ufw or iptables to restrict access to only the necessary IPs:

    ufw allow from <your_ip> to any port 9200
    ufw allow from <your_ip> to any port 5601
  • Set up an HTTPS reverse proxy: You can secure Kibana by setting up an HTTPS reverse proxy with Nginx: Set up an HTTPS reverse proxy with Nginx.

  • Set up TLS/SSL for Elasticsearch and Kibana: Ensure communications are encrypted between components using SSL/TLS as shown above.

Test the installationLink to this anchor

After completing the setup, you can verify if everything is working:

  • Elasticsearch: Run the following command to check Elasticsearch health:

    curl -X GET "localhost:9200/_cluster/health?pretty"
  • Kibana: Navigate to http://your_server_ip:5601 in your web browser.

  • Filebeat: Ensure logs are being shipped by checking the status:

    curl -X GET "localhost:5601/api/status"

Now, you should have a basic Elastic stack up and running! Adjust configurations as needed for your specific use case and further secure and optimize your setup for production use. Refer to the official Elastic documentation for the most accurate and up-to-date instructions and advanced configuration information.

Was this page helpful?
API DocsScaleway consoleDedibox consoleScaleway LearningScaleway.comPricingBlogCareers
© 2023-2025 – Scaleway