Fluentd Blog

Drop schedule announcement about EOL of Treasure Agent (td-agent) 4

Hi users and developers!

We announce the dropping schedule for TD Agent 4 development.

About Treasure Agent (td-agent) 4

Since td-agent 4.0.0 was released in 2020, td-agent 4 has been maintained for a long time.

As new major version of td-agent 4 was rebranded (to fluent-package v5), and released in August 2023, we decided to stop maintaining td-agent 4.

Scheduled end of life - Dec 2023

As you know, already stepping down maintenance activity, new minor update for td-agent 4 will not be shipped anymore. (td-agent v4.5.1 will be the last release of this series, it will be shipped in this month.)

Thus, we recommend using fluent-package v5 for new deployment :)

How to migrate to Fluent Package v5

There is a good article to do it.

• Upgrade to fluent-package v5

Follow the above instructions.

Read More

Upgrade to fluent-package v5

fluent-package "v5" is available since August 2023. fluent-package is the successor of td-agent "v4".

In this post, we will share the steps we've tested and hopefully this will help your experience from v4 to v5.

Why package was renamed?

fluent-package was formerly known as td-agent. In the past, Treasure Data, Inc took the initiative to provide the package, but now the Fluentd community does it. This is why the package name was changed.

To represent "All in one package of Fluentd which contains Fluentd and related gem packages", the package name was changed to fluent-package.

Even though package name was changed, Treasure Data, Inc still sponsor the package delivery resources.

Which channel should I use?

There are two channels for fluent-package.

• Normal release version
• Long Term Support version

One is normal release version which will be updated regularly - rapid release development style (td-agent v4 was released like this in almost every quarter). In this version, fluentd will be eventually updated to newer minor version (e.g. 1.17.x and so on)

The other is a more conservative maintenance version (Long Term Support) which will not introduce a new feature. It only applies teeny update such as security or bug fix only. LTS for v5 will be supported until March, 2025.

More details about difference between normal release version and LTS version will be explained in Scheduled support lifecycle announcement about Fluent Package.

Differences between td-agent v4 and fluent-package v5

In the fluent-package v5, core components like ruby (2.7.8 -> 3.2.2) and OpenSSL (1.1.1 -> 3.1.0 for Windows, 3.0.8 for macOS) were updated.

The major changes are as follows.

• td-agent command is renamed to fluentd.
  • $ td-agent --version -> $ fluentd --version
• td-agent-gem command is renamed to fluent-gem.
  • $ td-agent-gem list -> $ fluent-gem list
• The service name for non-Windows td-agent is renamed to fluentd.
  • $ systemctl status td-agent -> $ systemctl status fluentd

With the change of package name, install path, service name (e.g. /opt/fluent, fluentd.service) and so on were also changed. Basically, for td-agent v4 users, it aims to keep compatibility as far as possible by executing the migration process with copying old files or providing symbolic links for it.

If you want to know the details of upgraded components, see CHANGELOG.md.

NOTE: We explain for platform specific issue as "Additional hints for v4 users" section below.

Upgrade steps

During the upgrade process, plugins bundled in td-agent are automatically upgraded. With that being said, other plugins added on your own are not included. You should review if you need to upgrade plugins since some directory structures from v4 and v5 are changed.

In this post, I will show steps with plugins added on my own, "fluent-plugin-concat" for instance. Here is sample configuration file I used through steps.

<filter docker.log>
  @type concat
  key message
  multiline_start_regexp /^Start/
</filter>

1. Review what plugins are installed together with td-agent v4.

$ td-agent-gem list | grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)

You can also find installed plugins under /opt/td-agent/lib/ruby/gems/2.7.0/gems/ directories.

$ ls -l /opt/td-agent/lib/ruby/gems/2.7.0/gems |grep fluent-plugin*
drwxr-xr-x. 5 root root  175  7月 14 03:01 fluent-plugin-calyptia-monitoring-0.1.3
drwxr-xr-x. 4 root root  206  7月 14 03:03 fluent-plugin-concat-2.5.0
drwxr-xr-x. 5 root root 4096  7月 14 03:01 fluent-plugin-elasticsearch-5.3.0
drwxr-xr-x. 4 root root  205  7月 14 03:01 fluent-plugin-flowcounter-simple-0.1.0
drwxr-xr-x. 6 root root  191  7月 14 03:01 fluent-plugin-kafka-0.19.0
drwxr-xr-x. 5 root root  190  7月 14 03:01 fluent-plugin-metrics-cmetrics-0.1.2
drwxr-xr-x. 5 root root 4096  7月 14 03:01 fluent-plugin-opensearch-1.1.0
drwxr-xr-x. 5 root root  215  7月 14 03:01 fluent-plugin-prometheus-2.0.3
drwxr-xr-x. 6 root root  238  7月 14 03:01 fluent-plugin-prometheus_pushgateway-0.1.0
drwxr-xr-x. 4 root root  176  7月 14 03:01 fluent-plugin-record-modifier-2.1.1
drwxr-xr-x. 3 root root  210  7月 14 03:01 fluent-plugin-rewrite-tag-filter-2.4.0
drwxr-xr-x. 5 root root  230  7月 14 03:01 fluent-plugin-s3-1.7.2
drwxr-xr-x. 3 root root  170  7月 14 03:01 fluent-plugin-sd-dns-0.1.0
drwxr-xr-x. 3 root root   49  7月 14 03:01 fluent-plugin-systemd-1.0.5
drwxr-xr-x. 5 root root  221  7月 14 03:01 fluent-plugin-td-1.2.0
drwxr-xr-x. 5 root root  166  7月 14 03:01 fluent-plugin-utmpx-0.5.0
drwxr-xr-x. 4 root root  191  7月 14 03:01 fluent-plugin-webhdfs-1.5.0

2. Stop td-agent v4 daemon.

$ sudo systemctl stop td-agent

Even though fluent-package supports upgrade without stopping service, but recommend to stop explicitly.

3. Run installation script of fluent-package v5.

When you use RedHat or derivative distributions, you can run following script if you want to install normal release version of fluent-package.

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5.sh | sh

When you use RedHat or derivative distributions, you can run following script if you want to install LTS (Long term support) version of fluent-package.

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh

You can find more information about the installation script in Fluend Doc - Installation.

4. Confirm if fluent-package v5 is properly installed.

$ LANG=C yum info fluent-package
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.riken.jp
 * extras: ftp.riken.jp
 * updates: ftp.riken.jp
Installed Packages
Name        : fluent-package
Arch        : x86_64
Version     : 5.0.0
Release     : 1.el7
Size        : 64 M
Repo        : installed
From repo   : /fluent-package-5.0.0-1.el7.x86_64
Summary     : The stable distribution of Fluentd
URL         : https://www.treasuredata.com/
License     : ASL 2.0
Description : The stable distribution of Fluentd, called td-agent.

5. Reload fluent-package daemon.

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now fluentd

6. Check installed plugins.

$ fluent-gem list |grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)

You can see bundled plugins are upgraded as well but can not find plugins added on your own. In this post, added plugin was "fluent-plugin-concat" and it is not shown in installed list.

7. Install plugins added on my own.

$ sudo fluent-gem install fluent-plugin-concat

$ fluent-gem list | grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)

As for fluent-package v5, "fluent-plugin-concat" was installed under "/opt/fluent/lib/ruby/gems/3.2.0/gems/" directories.

$ ls -l /opt/fluent/lib/ruby/gems/3.2.0/gems/ |grep fluent-plugin*
drwxr-xr-x.  5 root root  175  7月 14 03:14 fluent-plugin-calyptia-monitoring-0.1.3
drwxr-xr-x.  4 root root  206  7月 14 03:16 fluent-plugin-concat-2.5.0
drwxr-xr-x.  5 root root 4096  7月 14 03:14 fluent-plugin-elasticsearch-5.3.0
drwxr-xr-x.  4 root root  205  7月 14 03:14 fluent-plugin-flowcounter-simple-0.1.0
drwxr-xr-x.  6 root root  191  7月 14 03:14 fluent-plugin-kafka-0.19.0
drwxr-xr-x.  5 root root  190  7月 14 03:14 fluent-plugin-metrics-cmetrics-0.1.2
drwxr-xr-x.  5 root root 4096  7月 14 03:14 fluent-plugin-opensearch-1.1.0
drwxr-xr-x.  5 root root  215  7月 14 03:14 fluent-plugin-prometheus-2.0.3
drwxr-xr-x.  6 root root  238  7月 14 03:14 fluent-plugin-prometheus_pushgateway-0.1.0
drwxr-xr-x.  4 root root  176  7月 14 03:14 fluent-plugin-record-modifier-2.1.1
drwxr-xr-x.  3 root root  210  7月 14 03:14 fluent-plugin-rewrite-tag-filter-2.4.0
drwxr-xr-x.  5 root root  230  7月 14 03:14 fluent-plugin-s3-1.7.2
drwxr-xr-x.  3 root root  170  7月 14 03:14 fluent-plugin-sd-dns-0.1.0
drwxr-xr-x.  3 root root   49  7月 14 03:14 fluent-plugin-systemd-1.0.5
drwxr-xr-x.  5 root root  221  7月 14 03:14 fluent-plugin-td-1.2.0
drwxr-xr-x.  5 root root  166  7月 14 03:14 fluent-plugin-utmpx-0.5.0
drwxr-xr-x.  4 root root  191  7月 14 03:14 fluent-plugin-webhdfs-1.5.0

8. Start fluent-package v5 daemon.

$ sudo systemctl start fluentd

9. Check if there are no error messages in fluentd logs.

$ tail -100f /var/log/fluent/fluentd.log

Now, upgrading steps are completed. Happy Logging!

Additional hints for v4 users

For Debian/Ubuntu

• fluentd-apt-source package will be marked as a transitional package. you can remove it safely with sudo apt purge fluentd-apt-source.
• If you want to enable td-agent.service, you must explicitly execute the following commands:

  $ sudo systemctl unmask td-agent
  $ sudo systemctl enable fluentd

For RHEL

• If you want to enable td-agent.service, you must explicitly execute the following command:

  $ sudo systemctl enable fluentd

For Windows

• fluent-package installer was changed not to start service by default. If you want to start fluentd as a service, execute the following command with administrator privileges.

  c:\opt\fluent> net start fluentdwinsvc

For macOS

WARNING: Currently we have no plan to officially support dmg version of fluent-package yet. It is just modified to be a minimally buildable state, it is for testing purpose only.

Read More

td-agent v4.5.0 has been released

Hi users!

We have released td-agent v4.5.0. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.4.2

• Update bundled Ruby to 2.7.8 which is the last version of Ruby 2.7 series.
  • Only for RHEL 9 & Ubuntu 22.04 jammy, we updat Ruby to 3.1.4 exceptionally.
• Update fluentd to 1.16.1
• Update plugins
  • fluent-plugin-kafka 0.19.0
  • fluent-plugin-elasticsearch 5.3.0
  • fluent-plugin-opensearch 1.1.0
• Drop Debian buster support

About next major version

We are preparing next major version of td-agent which bundles Ruby 3.2. It's still work in progerss, we'll announce it as soon as we are ready for it.

Please note that the package name will be changed to fluent-package. Please see the following discussion for more detail: https://github.com/fluent/fluentd/discussions/3860

Download

Please see the download page.

Read More

td-agent v4.4.2 has been released

Hi users!

We have released td-agent v4.4.2. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.4.1

• Update fluentd to 1.15.3
  • Fix log initializer to correctly create per-process files on Windows
  • out_file: Fix the multi-worker check with <worker 0-N> directive
  • Remove meaningless oj options
  • Support glob for !include directive in YAML config format
• macOS: Support Apple Silicon
• Update plugins
  • fluent-plugin-elasticsearch 5.2.4
  • fluent-plugin-opensearch 1.0.8
  • fluent-plugin-s3 1.7.2
  • fluent-plugin-windows-eventlog 0.8.2

Download

Please see the download page.

Read More

td-agent v4.4.1 has been released

Hi users!

We have released td-agent v4.4.1. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.4.0

• Update fluentd to 1.15.2
  • out_file: Fix append mode with --daemon flag
  • Drop win32-api gem dependency
  • Please see also the release announcement of v1.15.2
• Update fluent-plugin-kafka to 0.18.1
  • Fix outkafka2 doesn't respect `chunklimitrecordsandchunklimit_size`

Download

Please see the download page.

Read More

td-agent v4.4.0 has been released

Hi users!

We have released td-agent v4.4.0. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.3.2

• Update bundled Fluentd from v1.14.6 to v1.15.1. Please see each release announcement of Fluentd for more detail:
  • Fluentd v1.15.0 has been released
  • Fluentd v1.15.1 has been released
• Update OpenSSL to 1.1.1q (for macOS and Windows)
• Remove TD_AGENT_OPTIONS= from td-agent.service. Instead it's loaded from /etc/sysconfig/td-agent (rpm) or /etc/default/td-agent (deb).
• Drop Ubuntu 16.04 "Xenial Xerus" support as we announced before.
• Update plugins and gems

Please see ChangeLog for more detail.

Download

Please see the download page.

Read More

td-agent v4.3.2 has been released

We have released td-agent v4.3.2. This release is mainly for supporting new distributions released in 2022Q2. We choose Ruby 3.1 to bundle for such new distributions to adopt to recent environment such as OpenSSL 3.0. For other distributions, we still use Ruby 2.7. We'll migrate to Ruby 3 completely after Ruby 3.2 is released.

Changes from td-agent v4.3.1

• Support new distributions
  • Red Hat Enterprise Linux 9
  • Ubuntu 22.04 LTS (Jammy Jellyfish)
• Remove nokogiri from Linux and macOS packages
  • It was not used in actual by default on such environments.
• Update yajl-ruby to v1.4.3 to address vulnerability
• Update OpenSSL to 1.1.1p (for macOS and Windows)

Download

Please see the download page.

Read More

td-agent v4.3.1.1 for Windows has been released

We have released td-agent v4.3.1.1 for Windows. This version is a bug fix release of v4.3.1 only for Windows platform.

Changes from td-agent v4.3.1

• Update OpenSSL to v1.1.1.o-2 to fix memory leak: fluent-package-builder#374
• Update nokogiri to 1.13.6 to fix some vulnerabilities.

Download

Please see the download page.

Read More

td-agent v4.3.1 has been released

Hi users!

We have released td-agent v4.3.1. td-agent is a stable distribution package of Fluentd.

Changes from td-agent v4.3.0

• Update bundled Fluentd from v1.14.3 to v1.14.6. Please see each release announcement of Fluentd for more detail:
  • Fluentd v1.14.4 has been released
  • Fluentd v1.14.5 has been released
  • Fluentd v1.14.6 has been released
• Update bundled Ruby from v2.7.5 to v2.7.6
• Add some new plugins
  • fluent-plugin-opensearch
  • fluent-plugin-metrics-cmetrics
  • fluent-plugin-calyptia-monitoring
  • fluent-plugin-windows-exporter
• rpm: Remove .build-id directories

Please see ChangeLog for more detail.

Download

Please see the download page.

About packages for latest distributions

Although Ubuntu 22.04 LTS (Jammy Jellyfish) is already released, we don't yet provide the official td-agent package for it. An experimental package which bundles Ruby 3.1 is work in progress.

End of life announcement

• The package for Ubuntu xenial won't be provided from the next minor update (td-agent 4.4.0) because Ubuntu 16.04.x had already reached to End of Standard Support.
• td-agent 3 series won't be maintained anymore.

Read More

Treasure Agent (td-agent) 3 will not be maintained anymore

Hi users and developers!

We announce the dropping schedule for TD Agent 3 development.

About Treasure Agent (td-agent) 3

Since td-agent 3.0.0 was released in 2017, td-agent 3 had been maintained for a long time.

As new major version of td-agent 4 was released in August 2020, and several updates was shipped, we decided to stop maintaining td-agent 3.

Scheduled end of life - Feb 2022

As you know, already stepping down maintenance activity, new minor update for td-agent 3 will not be shipped anymore.

Thus, we recommend to use td-agent 4, latest fluentd v1 series, for new deployment :)

How to migrate to Treasure Agent (td-agent) 4

There is a good article to do it.

• Upgrade td-agent from v3 to v4

Follow the above instructions.

Read More