TOC

Fluentd Daemon Overview

Installation

• RPM Package (Redhat Linux)

  curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
  

• DEB Package (Debian / Ubuntu Linux), ubuntu 14.04 LTS / Trusty

  curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
  

Service commands of td-agent

• Launch daemon service td-agent start

• Reload configuration service td-agent reload

Locations

• Configuration file
  /etc/td-agent/td-agent.conf

• Plugin install command /usr/sbin/td-agent-gem install fluent-plugin-${PluginName}

• Plugin uninstall command /usr/sbin/td-agent-gem uninstall fluent-plugin-${PluginName}

• Logs /var/log/td-agent/td-agent.log

Configuration

Parser Plugin

Customer defined the field keys of the event record. Check Here for Built-in Parsers.

regexp - regexp for the format field

• Fluentular
• If the parameter value starts and ends with “/”, it is considered to be a regexp. TODO...

  format /^(?<logdt>[^|]*)\|(?<loglevel>[^|]*)\|(?<logbody>.*)$/`
  

none

• This will parse the line as-is with the key name “message”.

Plugins

Buffer Plugin

• Buffered Output Parameters

copy Output Plugin

fluent-plugin-copy_ex

Local File to BigQuery - BigQuery Output Plugin

Installation

td-agent-gem install fluent-plugin-bigquery

Configuration

Generates credential

td-agent.conf

• Bigquery Quota Policy

Local File to Google Storage - Google Cloud Storage Out

Installation

td-agent-gem install fluent-plugin-google-cloud-storage-out