suportebahia/Equipe SBahia - Elastic Engineer

Elastic Engineer - Equipe SBahia

Responsabilidades

Ingestão de Dados

• Configurar Beats (Filebeat, Metricbeat, Heartbeat, Auditbeat)
• Criar pipelines Logstash
• Integração com fontes (APIs, DBs, arquivos, syslog)
• Processamento e transformação
• TLS e autenticação

Elasticsearch

• Arquitetura de clusters
• Mapeamentos e settings
• Shards e réplicas
• Index lifecycle management (ILM)
• Otimização de queries
• Vector database

Kibana

• Dashboards e visualizações
• Lens e visualizations
• Canvas para reports
• Machine learning jobs
• Alerts e监测

Observability

• Logs centralizados
• Métricas de infraestrutura
• APM (Application Performance Monitoring)
• Uptime monitoring
• Tracing distribuído

Security

• SIEM - Security Information
• Elastic Security
• Threat hunting
• Detect rules
• Auditbeat

Competências Técnicas

Beats Configuration

# filebeat.yml - Coleta de logs
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/*.log
    fields:
      service: nginx
      environment: production
    multiline.pattern: '^\['
    multiline.negate: true
    multiline.match: after

output.logstash:
  hosts: ["logstash:5044"]
  ssl.enabled: true
  ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

# metricbeat.yml - Métricas do sistema
metricbeat.modules:
  - module: system
    metricsets:
      - cpu
      - memory
      - network
      - process
    period: 10s
  
  - module: nginx
    metricsets: ["status"]
    hosts: ["nginx:80"]
  
  - module: elasticsearch
    metricsets: ["node", "node_stats"]
    hosts: ["elasticsearch:9200"]
    period: 10s

Logstash Pipeline

# pipeline.conf
input {
  beats {
    port => 5044
  }
  
  jdbc {
    jdbc_connection_string => "jdbc:mysql://mysql:3306/orders"
    jdbc_user => "elastic"
    jdbc_password => "${MYSQL_PASSWORD}"
    statement => "SELECT * FROM orders WHERE updated_at > :sql_last_value"
    use_column_value => true
    tracking_column => "updated_at"
    schedule => "*/5 * * * *"
  }
}

filter {
  # Parse JSON
  json {
    source => "message"
    target => "parsed"
  }
  
  # Grok patterns para logs
  grok {
    match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{DATA:service} - %{GREEDYDATA:log}"
    }
  }
  
  # Date parsing
  date {
    match => ["timestamp", "ISO8601"]
    target => "@timestamp"
  }
  
  # Enrichment
  mutate {
    add_field => { "environment" => "production" }
  }
  
  # Remove fields
  mutate {
    remove_field => ["host", "agent"]
  }
}

output {
  elasticsearch {
    hosts => ["https://elasticsearch:9200"]
    index => "logs-%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    user => "elastic"
    password => "${ES_PASSWORD}"
    ssl => true
    ssl_certificate_verification => true
  }
}

Elasticsearch Index Template

{
  "index_patterns": ["logs-*", "metrics-*"],
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1,
      "index.lifecycle.name": "logs-policy",
      "index.mapping.total_fields.limit": 2000,
      "analysis": {
        "analyzer": {
          "log_analyzer": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "asciifolding"]
          }
        }
      }
    },
    "mappings": {
      "properties": {
        "timestamp": { "type": "date" },
        "message": { "type": "text", "analyzer": "log_analyzer" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "environment": { "type": "keyword" },
        "host": {
          "properties": {
            "name": { "type": "keyword" },
            "ip": { "type": "ip" },
            "os": { "type": "object" }
          }
        },
        "labels": { "type": "object", "dynamic": true },
        "geoip": {
          "properties": {
            "location": { "type": "geo_point" },
            "country": { "type": "keyword" }
          }
        }
      }
    }
  }
}

Query DSL

// Busca com filtros
GET logs-2024.01.15/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "service": "api" } },
        { "range": { "@timestamp": { "gte": "now-1h" } } }
      ],
      "filter": [
        { "term": { "level": "error" } }
      ],
      "should": [
        { "match_phrase": { "message": "connection timeout" } }
      ],
      "minimum_should_match": 1
    }
  },
  "aggs": {
    "errors_by_service": {
      "terms": { "field": "service", "size": 10 },
      "aggs": {
        "error_messages": {
          "terms": { "field": "message.keyword", "size": 5 }
        }
      }
    }
  },
  "highlight": {
    "fields": { "message": {} }
  }
}

Vector Search

from elasticsearch import Elasticsearch
from sklearn.feature_extraction.text import TfidfVectorizer

# Criar índice com mapping para vector
es.indices.create(
    index="documents",
    body={
        "mappings": {
            "properties": {
                "text": {"type": "text"},
                "text_vector": {
                    "type": "dense_vector",
                    "dims": 384,
                    "index": True,
                    "similarity": "cosine"
                }
            }
        }
    }
)

# Indexar documentos
documents = [
    {"text": "Python tutorial", "text_vector": vector1},
    {"text": "JavaScript guide", "text_vector": vector2},
]

for doc in documents:
    es.index(index="documents", body=doc)

# Semantic search
es.search(
    index="documents",
    body={
        "query": {
            "script_score": {
                "query": {"match_all": {}},
                "script": {
                    "source": "cosineSimilarity(params.query_vector, 'text_vector') + 1.0",
                    "params": {"query_vector": query_vector}
                }
            }
        }
    }
)

Kibana Dashboard JSON

{
  "title": "API Monitoring Dashboard",
  "panels": [
    {
      "id": "requests-over-time",
      "type": "line",
      "params": {
        "series": [
          {
            "id": "total-requests",
            "split_mode": "everything",
            "metrics": [{ "id": "metric", "type": "count" }]
          }
        ],
        "axis_formatter": "number",
        "interval": "auto"
      }
    },
    {
      "id": "response-codes",
      "type": "pie",
      "params": {
        "bucket": "response_code",
        "metric": "count"
      }
    },
    {
      "id": "latency-p95",
      "type": "gauge",
      "params": {
        "gauge": {
          "alignment": "vertical",
          "colors": ["green", "yellow", "red"],
          "max": 2000,
          "min": 0
        }
      }
    }
  ]
}

ILM (Index Lifecycle Management)

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "30d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "90d",
        "actions": {
          "freeze": {},
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "365d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Cluster Architecture

                         ┌─────────────┐
                         │   Kibana    │
                         └──────┬──────┘
                                │
         ┌──────────────────────┼──────────────────────┐
         │                      │                      │
┌────────▼────────┐    ┌────────▼────────┐    ┌────────▼────────┐
│  Logstash      │    │  Beats          │    │  Elasticsearch │
│  (Processing)  │    │  (Collection)   │    │  (Storage)     │
└────────┬────────┘    └────────┬────────┘    └───────┬────────┘
         │                      │                      │
         └──────────────────────┼──────────────────────┘
                                │
                    ┌───────────▼───────────┐
                    │   Elasticsearch       │
                    │   Cluster (3 nodes)   │
                    └───────────────────────┘

High Availability

# docker-compose.yml
version: '3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - cluster.name=prod-cluster
      - discovery.seed_hosts=elasticsearch
      - xpack.security.enabled=true
      - xpack.security.authc.anonymous.enabled=false
    volumes:
      - es-data:/usr/share/elasticsearch/data
    deploy:
      replicas: 3
    resources:
      limits:
        memory: 4G

  kibana:
    image: docker.elastic.co/kibana/kibana:8.11.0
    environment:
      - ELASTICSEARCH_HOSTS=https://elasticsearch:9200

  logstash:
    image: docker.elastic.co/logstash/logstash:8.11.0
    volumes:
      - ./pipeline:/usr/share/logstash/pipeline

Observability Stack

APM Setup

# apm-server.yml
apm-server:
  host: "0.0.0.0:8200"
  
  rum:
    enabled: true
    allow_origins: ['*']
  
  secret_token: "${APM_SECRET}"

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

Monitoramento

# heartbeat.yml - Uptime monitoring
heartbeat.monitors:
  - type: http
    id: api-health
    name: API Health Check
    urls: ["https://api.example.com/health"]
    schedule: '@every 30s'
    timeout: 10s
    check.response:
      status: 200

Performance Tuning

| Parameter | Default | Recommended | |-----------|---------|-------------| | number_of_shards | 1 | 3-5 por node | | number_of_replicas | 1 | 1-2 | | refresh_interval | 1s | -1 (disable) bulk | | translog.durability | async | request (critical) | | indices.memory.index_buffer_size | 10% | 20% |

Ferramentas

| Categoria | Ferramenta | |-----------|------------| | Collection | Beats (Filebeat, Metricbeat, Heartbeat, Packetbeat, Auditbeat) | | Processing | Logstash, Ingest Node Pipelines | | Storage | Elasticsearch | | Visualization | Kibana, Canvas, Lens | | Security | Elastic Security, SIEM | | Monitoring | APM, Uptime, Logs, Metrics | | Cloud | Elastic Cloud, ECE |

Boas Práticas

1. Index Lifecycle - Configure ILM desde o início
2. Replicas - Mínimo 1 para produção
3. Sharding - 50GB por shard ideal
4. Aliases - Use aliases para zero-downtime
5. Monitoring - cluster health monitoring
6. Backups - Snapshot/restore configurado
7. Security - TLS, RBAC, audit logging
8. Capacity Planning - Planeje crescimento