Timber!

Timber! Security Logging at UnitedHealth Group

@SerenaTiede

Who is Optum

Optum is the health services and innovation company within UnitedHealth Group

About Me

What Problem am I trying to solve?

Long log ingest times
Inconsistent Log Formats
Vendor lock in

Outline

Schema
Streaming
Pain Points and Tips

8 Required Fields

device.vendor
device.product
device.hostname
device.ip4
timeStamp
msg
application.id
application.name

couple notes

So a security event is something like a user logging in, what was query was ran on database, or what was the api call someone made

The required fields provide the bare minimum information that your security analysts need to determine what is and isn't malicious

device vendor would be your company if it's an in house app

device product is the name of the app

So for our in house stuff the vendor is Optum and for product let's say MyUhc

device hostname is the hostname but not localhost

device ipv4 we require it in integer format instead of the normal dotted decimal notation. This makes validation easier. An ip address sent cannot be 0.0.0.0 or 127.0.0.1

timestamp is milliseconds after since the unix epoch. We used to have a lot of non standard timestamp formats and sometimes there wouldn't be a timezone offset, and unix time sidesteps those issues

message is a human readable string describing what is happening in this event

Application.id We have an internal database that we use to keep tabs on all applications within UHG it's mainly used for funding and billing but since every application has a specific ID we kinda use that to keep tabs on who is logging to us.

similiar to application.id "application.name" is the name of your application within your CMDB (config management database)

Why You Want a Schema

It's hard to parse a variety of formats
Avro gives pretty much free type validation
Next to no need for data cleanup

now structured logging isn't anything new, but it takes time and specifically intent. when i'm writing my automation tooling i don't think about logging at all. the main thing a schema gives you have an easier time doing analytics

now given that we're using avro if a message is invalid it won't serialize correctly. this part is kinda a hard sell to application teams. From a security perspective a common schema makes the analytics easier because you've done the data cleanup before ingestion. With application logs one application can be way different than the next and writing a custom parser just doesn't scale. What the schema tries to do is try to make the data more valuable.

now please note i said data cleanup and not data quality! We get messages with the minimum fields that were better than the old stuff but we kinda need more info

How to Sell Your Schema

Change Policy
Give Feedback
Have an open Slack/Flowdock Channel
Be Patient

So one of the perks of being in security is that you can pretty easily change policies. But it's one thing to just say this is how developers need to log, but it can cause friction between development and security groups.

I'll have a screenshot in the next slide but give people a means to see if their logs meet the schema. Also we got this a lot "have you received our logs" so we have a nice grafana dashboard for that as well

Specifically at UHG we made a huge change to our logging policy. It takes a little while to get information out to my org's developer community. So we've have an open chat room where people can come in and ask us a questions. We also use github enterprise and we actively ask people to open issues or send us a pull request for our documentation.

Again be patient it takes time and energy to change your logging in your code. Also we've had to make some exceptions for people to try to meet in the middle. We have to get logs from mainframe applications and it takes a lot of time. We had COBOL developers implement custom string handling so their logs could be in JSON which is wild to me. A couple fun things I learned from mainframe learned

Log Validator

Grafana Customer View

Why Should You Stream Logs

get logs off of a compromised endpoint
a good way to lower your time to detect issues

Kafka

How do people actually send logs

filebeat agent via chef
the official logging java package
or any other method

I haven't actually talked about how people send us logs yet. So we have a couple options for everyone. You can send us logs straight to our kafka topic, or since chef is pretty heavily used at UHG you just add our filebeat cookbook to your server's run list. All people need to do is tell our attribute manager where to look for logs and make sure they are formatted correctly.

Also we're a pretty heavy Java shop so we actually had one of the internal teams (think contractors but internal) that build us a java package that handles creating log events and sending them to us. The package will handle converting an ip address from dotted decimal to an integer. It also handles making the receive time as defined as the milliseconds since the unix epoch. We'll have to revisit this in 2038 but it'll be fine ... probably

Then finally if you use another language or don't want to use filebeat you'll need to use something else. For some people supporting vendor apps they usually use something like fluentd to do some of the heavier parsing. For our platform logging we do all the parsing and delivery using fluent-bit since it's light weight and can be more easily deployed in our dmz environment.

Streams Apps

Lower latency than batch processing
Real time metrics on logs coming in
Streams run in kubernetes
3 types of streams (validation, elastic indexer, and hdfs writer)

Firewall Order of Operations

do any bootstrapping and non kafka ops
add node to firewall rules
add node to the kafka cluster
optional: rebalance partitions

now this is the pain points and tips part of the talk. Here is the obligatory do as I say and not as i did

so when running in your org's dmz environment or anything that requires messing with the firewalls is that network connectivity can cause issues. I wasn't aware of that fact so i added a broker to the cluster and figured everything would be fine ... except clients had connectivity issues because that broker was the leader of a partition of a newly created topic.

Also when operating in your dmz in my environment i can't talk to the public internet at all. So i have to host a webserver and package all my components (I'm talking node_exporter, kafka, zookeeper, and schema registry, it's a lot)

Infra and Change Management

Instrument All the Things!

This isn't a pain point but a lesson learned. This has come up in Liz Fong-Jones keynote is that if you don't build in observability you're gonna have a bad time. Currently we are collecting metrics from our streams, kafka brokers, schema registry servers, and zookeeper nodes. We're getting logs from kafka, zookeeper, and schema registry using our own pipeline which is kinda a mixed bag. For example if I mess up the cluster then logs aren't going to make it back. We're still working on getting tracing setup (if you have done this please bug me on twitter or after my talk)

Another thing we have been using is a heartbeat generator. So we have a couple servers that all they do every 5s with nanosecond precision is send a message through kafka, and we setup a watcher in elasticsearch. if we are short on logs then elasticsearch sends us an email and we can triage (most of the time it's just a message took a little too long to process)