fluent-package v5.2.0 has been released

Hi users!

We have released fluent-package v5.2.0. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This release is a new release of v5.2 series.

Changes from fluent-package v5.1.0

  • Support upgrade fluentd service with zero-downtime
  • Update ruby to 3.2.6
  • Update bundled Fluentd to v1.18.0
  • Update bundled gems overall
  • Solved possibility of DoS through "NoMemoryError"
  • msi: set GEM_HOME/GEM_PATH in fluentd.bat
  • deb: suppress service restart by needrestart
  • Fixed SIGABORT error with fluent-plugin-systemd

Support upgrade fluentd service with zero-downtime

Previously, when upgrading the fluent-package, you had to stop the fluentd service, install the new version of the fluent-package, then install any necessary plugins, and finally restart the fluentd service.

Starting from fluent-package v5.2.0, we have introduced the zero-downtime restart feature that automatically installs the required plugins during the upgrade process and allows you to upgrade Fluentd without interrupting the logging service.

Please refer to "Advisory for upgrading with zero-downtime".

Solved possibility of DoS through "NoMemoryError"

In the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.

When that message was sent, it might cause "NoMemoryError" on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.

Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled.

msi: set GEM_HOME/GEM_PATH in fluentd.bat

In the previous versions, fluentd.bat doesn't set GEMHOME/GEMPATH explicitly.

If users set custom GEM_HOME / GEM_PATH environment variables, the batch file will not work as expected. Therefore, this patch will set the appropriate values in the batch file.

deb: suppress service restart by needrestart

We have changed so that Fluentd does not restart when the needrestart package is installed. fluent-package v5.2.0 places /etc/needrestart/conf.d/50-fluent-package.conf.

Fixed SIGABORT error with fluent-plugin-systemd

In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.

Advisory for upgrading with zero-downtime

The zero-downtime restart feature can be configured by FLUENT_PACKAGE_SERVICE_RESTART environment variable. Please refer to following section for more details.

  • Add FLUENT_PACKAGE_SERVICE_RESTART environmental variable
    • This section explains FLUENT_PACKAGE_SERVICE_RESTART environment variable.
  • Automate Plugin Install for Update on Demand
    • This section explains the mechanism that installs the plugins automatically during the upgrade process.

Add FLUENT_PACKAGE_SERVICE_RESTART environmental variable

We have introduced FLUENT_PACKAGE_SERVICE_RESTART environment variable to configure the zero-downtime restart feature. The variable exists in the following file:

  • RPM: /etc/sysconfig/fluentd
  • DEB: /etc/default/fluentd

Example:

FLUENT_PACKAGE_OPTIONS=""
# Control method to upgrade service (auto/manual) restart
FLUENT_PACKAGE_SERVICE_RESTART=auto

auto (default)

The service automatically restarts with zero-downtime restart feature when all of the following conditions are met:

  1. The service was active before updating.
  2. The installed and upgrading versions are required 5.2.0 or higher (both sides must support this feature).

The plugins will be automatically reinstalled if needed if your environment has an active online connection. For more details, please refer to "Automate Plugin Install for Update on Demand".

manual

You can use this approach if you prefer to manage plugins manually or if your environment does not have offline connection. If the service was active before the update, it will not restart automatically with this configuration. You need to restart the service manually.

  • The zero-downtime restart:
    • Send a SIGUSR2 signal to the supervisor process, such as with kill -USR2 <PID>
  • Normal restart:
    • Restart the service normally, such as with systemctl restart fluentd.

Automate Plugin Install for Update on Demand

Automate Plugin Management

If you prefer to manage plugins automatically, please set FLUENT_PACKAGE_SERVICE_RESTART to auto (default).

When the service restarts automatically using the zero-downtime restart feature, any missing plugins are automatically detected and reinstalled before the restart.

Previously, automatic restarts after updates were not recommended due to potential issues. If you manually install plugins, you need to reinstall them before restarting when the embedded Ruby version is updated.

To address this, this feature enables automatic plugin installation during the restart process. The steps are as follows:

  1. Collect the current list of plugins before the update.
    • Recognize gems as plugins that has fluent-plugin- prefix.
  2. After installing the updated package, detect missing plugins.
    • Compare the collected plugin list with the default plugins to identify any missing ones.
  3. Install the missing plugins.

Manual Plugin Management

If you prefer to manage plugins manually, please set FLUENT_PACKAGE_SERVICE_RESTART to manual. This is useful in cases such as:

  • Pinning specific plugin versions
  • Operating in an offline environment

In this mode, automatic plugin installation and restarts are disabled. You can manually install plugins and send a SIGUSR2 signal to trigger the zero-downtime restart feature after the update.

Caution: if you use a custom unit file, need to migrate it to use this feature safely

If you use a custom unit file, such as /etc/systemd/system/fluentd.service, please remove these 2 lines.

Environment=GEM_HOME=/opt/fluentd/lib/ruby/gems/...
Environment=GEM_PATH=/opt/fluentd/lib/ruby/gems/...

We don't need GEM_HOME and GEM_PATH. They will be removed after v5.2.0, but if you use a custom unit file, you need to remove them manually. If these variables are set, the zero-downtime restart feature may not work correctly. It is because the new process inherits the original environment variables if using this feature.

Please refer to #713 for more details.

Download

Please see the download page.

About next LTS schedule

We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Read More

fluent-package v5.0.5 has been released

Hi users!

We have released fluent-package v5.0.5. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.6 and Fluentd was updated to 1.16.6. We recommend upgrading to fluent-package v5.0.5!

Changes from fluent-package v5.0.4

In this release, focused on updating bundled ones.

  • Update ruby to 3.2.6
  • Update fluentd to 1.16.6
  • Update bundled gems overall
  • Solved possibility of DoS through "NoMemoryError"
  • msi: set GEM_HOME/GEM_PATH in fluentd.bat
  • Fixed SIGABORT error with fluent-plugin-systemd

Solved possibility of DoS through "NoMemoryError"

In the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.

When that message was sent, it might cause "NoMemoryError" on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.

Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled into fluent-package v5.0.5.

msi: set GEM_HOME/GEM_PATH in fluentd.bat

In the previous versions, fluentd.bat doesn't set GEMHOME/GEMPATH explicitly.

If users set custom GEM_HOME / GEM_PATH environment variables, the batch file will not work as expected. Therefore, this patch will set the appropriate values in the batch file.

Above bug was also fixed in fluent-package v5.0.5.

Fixed SIGABORT error with fluent-plugin-systemd

In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.

Download

Please see the download page.

About next LTS schedule

We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Read More

Scheduled support lifecycle announcement about Fluent Package v6

Hi users!

We had launched fluent-package v5 series last year, recently shipped v5.0.4 in LTS release channel and v5.1.0 in normal release channel.

In this blog article, we explains the planned next major updates - v6.0.0.

When the next LTS (Long Term Support) version available?

In short, we will ship fluent-package v6 in Aug, 2025.

We keeps two release channels as follows:

  • Normal release
  • LTS (Long Term Support)

Here is the difference of these channels.

  • Normal release (6.x series)
    • Recommended for early adopters (eager to try new features of Fluentd)
    • Rapid release development (planned semi-annually releases, but it may vary.)
    • Fluentd will be upgraded occasionally
      • Minor upgrade will be applied. e.g. 1.18 => 1.19 and so on.
      • e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), ... v6.1.0 (Fluentd v1.19.0)
    • The only latest version will be supported (bug fix will not be backported into older version)
  • LTS (Long Term Support, 6.0.x series)
    • Recommended for enterprise services
    • Security and bug fix release only
    • Fluentd will be upgraded only in the teeny version and will stick to 1.18.x series. (T.B.D.)
      • Thus, the version number will be 6.0.x. (6.1 series will not be shipped for LTS channel)
      • e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), v6.0.3 (Fluentd v1.18.3) ... v6.0.x (Fluentd v1.18.x) and so on.
    • Next major version will be fluent-package v7.0.0, and will be shipped in 2027 (T.B.D.).

As for fluent-package v6,

  • fluent-package v5 LTS will be supported until Dec, 2025.
    • We expect users upgrade from v5 to v6 during the grace period.
  • Until 6.1.0 is released, normal release channel and LTS channel are the same package.
  • Upgrade from v4 or older version is not supported. Upgrade to v5 first, then upgrade it to v6.

Current releases and future plans:

Release Version Release Date End of life
v5.0.x (LTS) Aug, 2023 Dec, 2025
v6.0.0 (LTS) Aug, 2025 Dec, 2027 (T.B.D.)
v7.0.0 (LTS) Aug, 2027 (T.B.D.) T.B.D.

Happy logging!

Read More

fluent-package v5.1.0 has been released

Hi users!

We have released fluent-package v5.1.0. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This release is a new release of v5.1 series. Bundled ruby version was upgraded to 3.2.5 and fluentd v1.17.0 was shipped!

Changes from fluent-package v5.0.4

In this release, focused on updating bundled ones.

  • Update fluentd to 1.17.0
  • Update ruby to 3.2.5
  • Update bundled gems overall

Update bundled components overall

In LTS version, conservative updating policy is enabled, so updating gem is very limited to bug fix or security fix. In contrast to LTS version, v5.1.0 bundles more recent components because of standard version.

As we already announced in Scheduled support lifecycle announcement about Fluent Package blog article, in normal release channel, we will ship the latest version (v1.17.0) of Fluentd.

So if you want to try using latest version of Fluentd, v5.1.0 is one for you. If you use fluent-package in enterprise services, keep using fluent-package 5.0.x (LTS).

Download

Please see the download page.

About next LTS schedule

We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Read More

fluent-package v5.0.4 has been released

Hi users!

We have released fluent-package v5.0.4. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.4 and a foolproof mechanism was implemented to prevent launching duplicated Fluentd instances, we recommend upgrading to fluent-package v5.0.4!

Changes from fluent-package v5.0.3

  • Update ruby to 3.2.4 (#645)
  • Fixed to prevent launching Fluentd wrongly if the service is already running (#648,#649)
  • Added support for Ubuntu 24.04 (Noble Numbat)

Fixed to prevent launching Fluentd wrongly if the service is already running

In this release, a foolproof mechanism was implemented to prevent launching Fluentd wrongly if the service is already running.

As you know, you can check the version of Fluentd with fluentd --version, but there is a case that fluentd -v is executed wrongly to do it.

When already running Fluentd as a service, fluentd -v launches a duplicated Fluentd instance with the same fluentd configuration. If you launch duplicated Fluentd instances, it causes the corruption of processing Fluentd buffer. To prevent such a situation, a foolproof was implemented now.

For example, if Fluentd is running as a service, launching Fluentd causes an error to block it.

Here is the example on Windows:

> fluentd
Error: Can't start duplicate Fluentd instance with the default config.

To start Fluentd, please do one of the following:
  (Caution: Please be careful not to start multiple instances with the same config.)
  - Stop the Fluentd Windows service 'fluentdwinsvc'.
  - Specify the config path explicitly by '-c' ('--config').

Even though if you wrongly launch Fluentd to check version with -v (It should be --version to show version), then it causes the following error.

> fluentd -v
Error: Can't start duplicate Fluentd instance with the default config.

To take the version, please use '--version', not '-v' ('--verbose').

To start Fluentd, please do one of the following:
  (Caution: Please be careful not to start multiple instances with the same config.)
  - Stop the Fluentd Windows service 'fluentdwinsvc'.
  - Specify the config path explicitly by '-c' ('--config')

Note that this foolproof feature is intended to block launching duplicated Fluentd instance, you can explicitly launch Fluentd by specifying a specific option to pass it even though already Fluentd is running as a service.

On Windows:

  • -c (--config)
  • --dry-run
  • --reg-winsvc
  • --reg-winsvc-fluentdopt
  • --show-plugin-config

On Linux:

  • -c (--config)
  • --dry-run
  • --show-plugin-config

About next LTS schedule

We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Download

Please see the download page.

Read More

fluent-package v5.0.3 has been released

Hi users!

We have released fluent-package v5.0.3. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!

Changes from fluent-package v5.0.2

  • Update fluentd to 1.16.5. See the following blog articles about details.
  • Update bundled plugins
    • e.g. fluent-diagtool v1.0.5. It supports to collect list of plugins on Windows.
  • msi: fixed wrong environment path for Fluent Package Prompt (#606)
    • It breaks fluent-diagtool behavior to launch fluent-gem correctly.
  • msi: removed unnecessary path delimiter (#607)
    • It doesn't cause any problem yet, but it should treat %~dp0 correctly.
  • rpm: fixed to take over enabled state of systemd service from td-agent v4 (#613)
  • deb rpm: fixed to quote target files correctly not to cause migration failures (#615)
  • msi: added a patch for RubyInstaller to avoid crash on start up (#620)
  • msi: fixed slow start issue on Windows (#631)

About next LTS schedule

We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Download

Please see the download page.

Read More

Open Source Summit Japan 2023

Hi users and developers!

We would like to share the information about an event appearance of Fluentd maintainers.

Daijiro Fukuda and Kentaro Hayashi, Fluentd Maintainers from ClearCode Inc., are going to talk at the Open Source Summit Japan on December 5th

The event has been taken thousands of attendees all over the world and one of the leading Open Source event in Japan. We are honored to take this opportunity and very excited to share about Fluentd project at the conference!

Everyone attending the event, please join us!

About the Talk

Date: December 5th 14:00-14:40
Speakers: Daijiro Fukuda and Kentaro Hayashi from ClearCode Inc.
Summary:
Fluentd, the CNCF graduated project, starts a new chapter as the OSS project. Fluentd community released the new package fluent-package as the successor of td-agent and started providing a new release cycle to fit better with needs. This talk is going to share the latest news of Fluentd, including its updates, package, rebranding and so on, with people who are using Fluentd or struggling to manage their logs. Fluentd is an open source data collector that unifies data collection and consumption. Because of its flexible plugin system and scalability, Fluentd is adopted in various systems and has become a de facto standard, especially for scale logging in distributed systems.

In the new release cycle, the package applies only bug fixes and security fixes of Fluentd over a long term, about two years. We are confident that this will meet the demands of users who want to use Fluentd stably over the long term. In this session, we will talk about (a) the history of Fluentd v1 and td-agent v4, (b) why Fluentd community decided to rebrand existing td-agent to fluent-package, (c) what the benefits and impacts of the new release cycle are, and (d) how to upgrade to fluent-package from td-agent.

Detail info: Link to event page

Overall Event Information

Name: Open Source Summit Japan 2023
Date: 2023-12-04 to 2023-12-06
Host: The Linux Foundation
Description:

Open Source Summit Japan is the leading conference in Japan connecting the open source ecosystem under one roof, providing a forum for technologists and open source industry leaders to collaborate and share information, learn about the latest in open source technologies and find out how to gain a competitive advantage by using innovative open solutions.

Open Source Summit is a conference umbrella, composed of a collection of events covering the most important technologies, topics, and issues affecting open source today.

Quote from the event web page : https://events.linuxfoundation.org/open-source-summit-japan/

Read More

fluent-package v5.0.2 has been released

Hi users!

We have released fluent-package v5.0.2 and td-agent v4.5.2. fluent-package is a stable distribution package of Fluentd.

This is a maintenance release of v5.0.x LTS series. As significant in_tail bugs (wrongly stopping tailing logs) were fixed in latest release, we recommend upgrading to fluent-package v5.0.2!

Changes from fluent-package v5.0.1

  • Update fluentd to 1.16.3 which contains significant bug fixes about in_tail. See Fluentd v1.16.3 and v1.16.2 have been released blog article about details.
  • Update plugins
    • fluent-diagtool v1.0.3. It supports fluent-package and can collect information about locally installed gems. It may help to migrate from td-agent v4 a bit. See Upgrade to fluent-package v5 for migration.
  • msi: support path which contains space or parenthesis (#589)
  • deb: fixed system user/group name in logrotate config (#592,#594)
    • It fixes a bug that unknown user error was reported during log rotation.
  • rpm: fixed to create fluentd user as system account (#596)
    • It fixes a bug that /var/lib/fluent directory was created unexpectedly. It doesn't affect the fluentd service behavior, but it is desirable one.
  • rpm: changed to keep system account after removing fluent-package. (#598)
    • In the previous versions, there was a bug that group was not cleanly removed when the package was upgraded from td-agent v4. This change makes reinstall/downgrade friendly.

About next LTS schedule

We plan to ship next LTS version of fluent-package v5.0.3 on Feb, 2024. The content of updates are still in T.B.D.

About td-agent v4.5.2

This is a exceptional maintenance release of v4.5.x series. Fluentd was updated to 1.16.3 because it contains significant bug fixes about in_tail. Note that td-agent will not be updated anymore. See Drop schedule announcement about EOL of Treasure Agent (td-agent) 4.

Download

Please see the download page.

Read More

Scheduled support lifecycle announcement about Fluent Package

Hi users!

We have released fluent-package v5.0.0 (Release Candidate) and the following v5.0.1 (General Availability). fluent-package is formerly known as td-agent and a stable distribution package of Fluentd.

In this blog article, we explains why the concept of channels is introduced.

Why is LTS (Long Term Support) channel introduced?

In the previous versions, td-agent was shipped as quarterly releases in most cases. It sometimes contains fluentd minor upgrade occasionally, which means that not only bug or security fixes, but also contains new features. As a result, there was a case that the stability was affected unexpectedly.

In point view of adopting fluentd (and td-agent) for enterprise services, there was a requirement for the distribution which contains security or bug fixes only and scheduled lifecycle of it.

The LTS release is the answer to support them.

Channels for updates

There are two channels of support lifecycle.

  • Normal release
  • LTS (Long Term Support)

Here is the difference of these channels.

  • Normal release (5.x series)
    • Recommended for early adopters (eager to try new features of Fluentd)
    • Rapid release development (planned semi-annually releases, but it may vary.)
    • Fluentd will be upgraded occasionally
      • Minor upgrade will be applied. e.g. 1.16 => 1.17 and so on.
      • e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), ... v5.1.0 (Fluentd v1.17.0)
    • The only latest version will be supported (bug fix will not be backported into older version)
  • LTS (Long Term Support, 5.0.x series)
    • Recommended for enterprise services
    • Security and bug fix release only
    • Fluentd will be upgraded only in the teeny version and will stick to 1.16.x series.
      • Thus, the version number will be 5.0.x. (5.1 series will not be shipped)
      • e.g. v5.0.1 (Fluentd v1.16.2), v5.0.2 (Fluentd v1.16.3), v5.0.3 (Fluentd v1.16.4) ... v5.0.x (Fluentd v1.16.x)
    • Next major version will be fluent-package v6.0.0, and will be shipped in 2025.

As for fluent-package v5,

  • fluent-package v5 LTS will be supported until Dec, 2025 (Even though Ruby 3.2 is still supported)
  • Until 5.1.0 is released, normal release channel and LTS channel are the same package.
Release Version Release Date End of life
v4.5.1 Aug, 2023 Dec, 2023
v5.0.1 (LTS) Aug 29, 2023 Dec, 2025
v6.0.0 (LTS) Aug, 2025? (T.B.D.) T.B.D.

Happy logging!

Read More

Upgrade to fluent-package v5

fluent-package "v5" is available since August 2023. fluent-package is the successor of td-agent "v4".

In this post, we will share the steps we've tested and hopefully this will help your experience from v4 to v5.

Why package was renamed?

fluent-package was formerly known as td-agent. In the past, Treasure Data, Inc took the initiative to provide the package, but now the Fluentd community does it. This is why the package name was changed.

To represent "All in one package of Fluentd which contains Fluentd and related gem packages", the package name was changed to fluent-package.

Even though package name was changed, Treasure Data, Inc still sponsor the package delivery resources.

Which channel should I use?

There are two channels for fluent-package.

  • Normal release version
  • Long Term Support version

One is normal release version which will be updated regularly - rapid release development style (td-agent v4 was released like this in almost every quarter). In this version, fluentd will be eventually updated to newer minor version (e.g. 1.17.x and so on)

The other is a more conservative maintenance version (Long Term Support) which will not introduce a new feature. It only applies teeny update such as security or bug fix only. LTS for v5 will be supported until March, 2025.

More details about difference between normal release version and LTS version will be explained in Scheduled support lifecycle announcement about Fluent Package.

Differences between td-agent v4 and fluent-package v5

In the fluent-package v5, core components like ruby (2.7.8 -> 3.2.2) and OpenSSL (1.1.1 -> 3.1.0 for Windows, 3.0.8 for macOS) were updated.

The major changes are as follows.

  • td-agent command is renamed to fluentd.
    • $ td-agent --version -> $ fluentd --version
  • td-agent-gem command is renamed to fluent-gem.
    • $ td-agent-gem list -> $ fluent-gem list
  • The service name for non-Windows td-agent is renamed to fluentd.
    • $ systemctl status td-agent -> $ systemctl status fluentd

With the change of package name, install path, service name (e.g. /opt/fluent, fluentd.service) and so on were also changed. Basically, for td-agent v4 users, it aims to keep compatibility as far as possible by executing the migration process with copying old files or providing symbolic links for it.

If you want to know the details of upgraded components, see CHANGELOG.md.

NOTE: We explain for platform specific issue as "Additional hints for v4 users" section below.

Upgrade steps

During the upgrade process, plugins bundled in td-agent are automatically upgraded. With that being said, other plugins added on your own are not included. You should review if you need to upgrade plugins since some directory structures from v4 and v5 are changed.

In this post, I will show steps with plugins added on my own, "fluent-plugin-concat" for instance. Here is sample configuration file I used through steps.

<filter docker.log>
  @type concat
  key message
  multiline_start_regexp /^Start/
</filter>

1. Review what plugins are installed together with td-agent v4.

$ td-agent-gem list | grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)

You can also find installed plugins under /opt/td-agent/lib/ruby/gems/2.7.0/gems/ directories.

$ ls -l /opt/td-agent/lib/ruby/gems/2.7.0/gems |grep fluent-plugin*
drwxr-xr-x. 5 root root  175  7月 14 03:01 fluent-plugin-calyptia-monitoring-0.1.3
drwxr-xr-x. 4 root root  206  7月 14 03:03 fluent-plugin-concat-2.5.0
drwxr-xr-x. 5 root root 4096  7月 14 03:01 fluent-plugin-elasticsearch-5.3.0
drwxr-xr-x. 4 root root  205  7月 14 03:01 fluent-plugin-flowcounter-simple-0.1.0
drwxr-xr-x. 6 root root  191  7月 14 03:01 fluent-plugin-kafka-0.19.0
drwxr-xr-x. 5 root root  190  7月 14 03:01 fluent-plugin-metrics-cmetrics-0.1.2
drwxr-xr-x. 5 root root 4096  7月 14 03:01 fluent-plugin-opensearch-1.1.0
drwxr-xr-x. 5 root root  215  7月 14 03:01 fluent-plugin-prometheus-2.0.3
drwxr-xr-x. 6 root root  238  7月 14 03:01 fluent-plugin-prometheus_pushgateway-0.1.0
drwxr-xr-x. 4 root root  176  7月 14 03:01 fluent-plugin-record-modifier-2.1.1
drwxr-xr-x. 3 root root  210  7月 14 03:01 fluent-plugin-rewrite-tag-filter-2.4.0
drwxr-xr-x. 5 root root  230  7月 14 03:01 fluent-plugin-s3-1.7.2
drwxr-xr-x. 3 root root  170  7月 14 03:01 fluent-plugin-sd-dns-0.1.0
drwxr-xr-x. 3 root root   49  7月 14 03:01 fluent-plugin-systemd-1.0.5
drwxr-xr-x. 5 root root  221  7月 14 03:01 fluent-plugin-td-1.2.0
drwxr-xr-x. 5 root root  166  7月 14 03:01 fluent-plugin-utmpx-0.5.0
drwxr-xr-x. 4 root root  191  7月 14 03:01 fluent-plugin-webhdfs-1.5.0

2. Stop td-agent v4 daemon.

$ sudo systemctl stop td-agent

Even though fluent-package supports upgrade without stopping service, but recommend to stop explicitly.

3. Run installation script of fluent-package v5.

When you use RedHat or derivative distributions, you can run following script if you want to install normal release version of fluent-package.

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5.sh | sh

When you use RedHat or derivative distributions, you can run following script if you want to install LTS (Long term support) version of fluent-package.

# curl -L https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh

You can find more information about the installation script in Fluend Doc - Installation.

4. Confirm if fluent-package v5 is properly installed.

$ LANG=C yum info fluent-package
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.riken.jp
 * extras: ftp.riken.jp
 * updates: ftp.riken.jp
Installed Packages
Name        : fluent-package
Arch        : x86_64
Version     : 5.0.0
Release     : 1.el7
Size        : 64 M
Repo        : installed
From repo   : /fluent-package-5.0.0-1.el7.x86_64
Summary     : The stable distribution of Fluentd
URL         : https://www.treasuredata.com/
License     : ASL 2.0
Description : The stable distribution of Fluentd, called td-agent.

5. Reload fluent-package daemon.

$ sudo systemctl daemon-reload
$ sudo systemctl enable --now fluentd

6. Check installed plugins.

$ fluent-gem list |grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)

You can see bundled plugins are upgraded as well but can not find plugins added on your own. In this post, added plugin was "fluent-plugin-concat" and it is not shown in installed list.

7. Install plugins added on my own.

$ sudo fluent-gem install fluent-plugin-concat
$ fluent-gem list | grep fluent-plugin*
fluent-plugin-calyptia-monitoring (0.1.3)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.3.0)
fluent-plugin-flowcounter-simple (0.1.0)
fluent-plugin-kafka (0.19.0)
fluent-plugin-metrics-cmetrics (0.1.2)
fluent-plugin-opensearch (1.1.0)
fluent-plugin-prometheus (2.0.3)
fluent-plugin-prometheus_pushgateway (0.1.0)
fluent-plugin-record-modifier (2.1.1)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluent-plugin-s3 (1.7.2)
fluent-plugin-sd-dns (0.1.0)
fluent-plugin-systemd (1.0.5)
fluent-plugin-td (1.2.0)
fluent-plugin-utmpx (0.5.0)
fluent-plugin-webhdfs (1.5.0)

As for fluent-package v5, "fluent-plugin-concat" was installed under "/opt/fluent/lib/ruby/gems/3.2.0/gems/" directories.

$ ls -l /opt/fluent/lib/ruby/gems/3.2.0/gems/ |grep fluent-plugin*
drwxr-xr-x.  5 root root  175  7月 14 03:14 fluent-plugin-calyptia-monitoring-0.1.3
drwxr-xr-x.  4 root root  206  7月 14 03:16 fluent-plugin-concat-2.5.0
drwxr-xr-x.  5 root root 4096  7月 14 03:14 fluent-plugin-elasticsearch-5.3.0
drwxr-xr-x.  4 root root  205  7月 14 03:14 fluent-plugin-flowcounter-simple-0.1.0
drwxr-xr-x.  6 root root  191  7月 14 03:14 fluent-plugin-kafka-0.19.0
drwxr-xr-x.  5 root root  190  7月 14 03:14 fluent-plugin-metrics-cmetrics-0.1.2
drwxr-xr-x.  5 root root 4096  7月 14 03:14 fluent-plugin-opensearch-1.1.0
drwxr-xr-x.  5 root root  215  7月 14 03:14 fluent-plugin-prometheus-2.0.3
drwxr-xr-x.  6 root root  238  7月 14 03:14 fluent-plugin-prometheus_pushgateway-0.1.0
drwxr-xr-x.  4 root root  176  7月 14 03:14 fluent-plugin-record-modifier-2.1.1
drwxr-xr-x.  3 root root  210  7月 14 03:14 fluent-plugin-rewrite-tag-filter-2.4.0
drwxr-xr-x.  5 root root  230  7月 14 03:14 fluent-plugin-s3-1.7.2
drwxr-xr-x.  3 root root  170  7月 14 03:14 fluent-plugin-sd-dns-0.1.0
drwxr-xr-x.  3 root root   49  7月 14 03:14 fluent-plugin-systemd-1.0.5
drwxr-xr-x.  5 root root  221  7月 14 03:14 fluent-plugin-td-1.2.0
drwxr-xr-x.  5 root root  166  7月 14 03:14 fluent-plugin-utmpx-0.5.0
drwxr-xr-x.  4 root root  191  7月 14 03:14 fluent-plugin-webhdfs-1.5.0

8. Start fluent-package v5 daemon.

$ sudo systemctl start fluentd

9. Check if there are no error messages in fluentd logs.

$ tail -100f /var/log/fluent/fluentd.log

Now, upgrading steps are completed. Happy Logging!

Additional hints for v4 users

For Debian/Ubuntu

  • fluentd-apt-source package will be marked as a transitional package. you can remove it safely with sudo apt purge fluentd-apt-source.
  • If you want to enable td-agent.service, you must explicitly execute the following commands:
  $ sudo systemctl unmask td-agent
  $ sudo systemctl enable fluentd

For RHEL

  • If you want to enable td-agent.service, you must explicitly execute the following command:
  $ sudo systemctl enable fluentd

For Windows

  • fluent-package installer was changed not to start service by default. If you want to start fluentd as a service, execute the following command with administrator privileges.
  c:\opt\fluent> net start fluentdwinsvc

For macOS

WARNING: Currently we have no plan to officially support dmg version of fluent-package yet. It is just modified to be a minimally buildable state, it is for testing purpose only.

Read More