Aggregated Log Source Ingestion a.k.a. ALSI (Cribl LogStream) Notes

THIS IS A WORK IN PROGRESS

ℹ️ The following guide should get you, dear reader, 90% there. Cribl does not lend itself well to orchestration so the Salt states may fail. They worked once upon a time and efforts have been made to keep them working but it may be faster to stand up Cribl manually; the choice is yours.

Create Okta (OIDC) Application

⚠️ This step can be skipped if there is no Enterprise license available.

Follow the instructions from docs.cribl.io to create the Okta application.

Prerequisites:

The Leader/Master ALB URL for the Web UI

You may be thinking, "What is the URL? I have not created it yet." and you are correct. Fortunately, we have a standard naming convention so it should be simple to enter the correct value (the value doesn't have to be correct right now, for what it is worth).

Example: https://<customer>-alsi.pvt.accenturefederalcyber.com

The base URL will be required in two places with URL suffixes found in the documentation. You can also compare against existing Cribl Stream Okta applications.

Whether or not you assign roles/users/etc. is not required at this point. We add the app to Okta first in order to get two strings to add to Salt.

Copy the Okta client ID and client secret. You will add them to the customer's pillar variables file as described below.

Add Cribl Pillars to Customer Variables SLS

Add the `mdr_wildcard_cert` pillars to Salt's `pillar/top.sls` for the ALSI instances.

# pillar/top.sls
---
### Find the customer's section and add
'<customer>-alsi-*':
  - mdr_wildcard_cert
### etc. etc.
...

Enable GPG

Add #!jinja|yaml|gpg to the top of the customer variables SLS file if it is not present.

Copy the cribl pillars from an existing customer such as Moose (moose_variables.sls) to the customer's variables file.

Replace all encrypted values except the admin_password with the appropriate GPG-encrypted blocks.

{% if grains['id'].startswith('bas-alsi-') %}
cribl:
  privatekey_path: "/opt/cribl/pki/privatekey.pem"
  certificate_path: "/opt/cribl/pki/cert.pem"
  # [] represents the free license distributed with Cribl.
  # The free license does not permit SSO.
  license: []
  hec_token: ~
  admin_password: |
    -----BEGIN PGP MESSAGE-----

    Value removed
    -----END PGP MESSAGE-----
  okta_client_id: ~
  okta_client_secret: ~
{% endif %} {# If alsi #}

:information_source: See the GnuPG (gpg) Notes document for instructions on how to GPG-encrypt the various values.

Creating the Cribl Infrastructure for a Customer

Copy the 175-splunk-alsi directory from an existing customer (or test Moose) to the customer's directory in the xdr-terraform-live repository.

cp -a ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-bas/175-splunk-alsi ~/xdr-terraform-live/prod/aws-us-gov/mdr-prod-<customer>/

Create Worker nodes

To create one or more worker nodes along with the leader, modify xdr-terraform-modules/base/splunk_servers/alsi/config.sls and set the number of workers as an exception, using the account name found in account.hcl.

ℹ️ Changing this value will create worker nodes in the Cribl VPC, not the LCP. Leave the number at zero (0) if the worker nodes will be in the LCP VPC(s).

  # If cribl is being used for log ingestion, remember to turn on splunk_private_hec
  # in `splunk_servers/indexer_cluster/config.tf`, too.
  alsi_workers_default = 0 # how many cribl workers
  alsi_workers_exceptions = {
    afs-mdr-test-c2-gov = 2,
    mdr-prod-bas        = 2,
  }

If the customer requires public ELBs for HEC, enable those in config.tf as well.

Enable Indexer Discovery for Cribl

If the Cribl worker nodes are in the customer slice (not in the LCP VPC), use Indexer Discovery to configure Cribl to send data to the XDR-hosted indexers. If the worker nodes are in the LCP, they can use the NLB URL.

Log into the customer's Cluster Manager (CM), go to Settings -> Tokens and click on the Token Settings button. Ensure Token Authentication is enabled and set the Default Expiration to 'never', then create an authentication token for the admin user with no expiration date (leave the date fields blank). Copy the token it creates before clicking the Close button.

Log into the customer's Cribl Leader as the admin user and configure a Stream destination to use a Splunk Load Balanced destination with indexer discovery enabled and provide the token. Be sure to enable TLS, otherwise Cribl will complain about reset connections.

ALSI (Cribl LogStream) Notes.md 4.6 KB 히스토리 Raw