Fluentd Blog

fluent-package v5.2.0 has been released

Hi users!

We have released fluent-package v5.2.0. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This release is a new release of v5.2 series.

Changes from fluent-package v5.1.0

• Support upgrade fluentd service with zero-downtime
• Update ruby to 3.2.6
• Update bundled Fluentd to v1.18.0
• Update bundled gems overall
• Solved possibility of DoS through "NoMemoryError"
• msi: set GEM_HOME/GEM_PATH in fluentd.bat
• deb: suppress service restart by needrestart
• Fixed SIGABORT error with fluent-plugin-systemd

Support upgrade fluentd service with zero-downtime

Previously, when upgrading the fluent-package, you had to stop the fluentd service, install the new version of the fluent-package, then install any necessary plugins, and finally restart the fluentd service.

Starting from fluent-package v5.2.0, we have introduced the zero-downtime restart feature that automatically installs the required plugins during the upgrade process and allows you to upgrade Fluentd without interrupting the logging service.

Please refer to "Advisory for upgrading with zero-downtime".

Solved possibility of DoS through "NoMemoryError"

In the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.

When that message was sent, it might cause "NoMemoryError" on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.

Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled.

msi: set GEM_HOME/GEM_PATH in fluentd.bat

In the previous versions, fluentd.bat doesn't set GEMHOME/GEMPATH explicitly.

If users set custom GEM_HOME / GEM_PATH environment variables, the batch file will not work as expected. Therefore, this patch will set the appropriate values in the batch file.

deb: suppress service restart by needrestart

We have changed so that Fluentd does not restart when the needrestart package is installed. fluent-package v5.2.0 places /etc/needrestart/conf.d/50-fluent-package.conf.

Fixed SIGABORT error with fluent-plugin-systemd

In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.

Advisory for upgrading with zero-downtime

The zero-downtime restart feature can be configured by FLUENT_PACKAGE_SERVICE_RESTART environment variable. Please refer to following section for more details.

• Add FLUENT_PACKAGE_SERVICE_RESTART environmental variable
  • This section explains FLUENT_PACKAGE_SERVICE_RESTART environment variable.
• Automate Plugin Install for Update on Demand
  • This section explains the mechanism that installs the plugins automatically during the upgrade process.

Add FLUENT_PACKAGE_SERVICE_RESTART environmental variable

We have introduced FLUENT_PACKAGE_SERVICE_RESTART environment variable to configure the zero-downtime restart feature. The variable exists in the following file:

• RPM: /etc/sysconfig/fluentd
• DEB: /etc/default/fluentd

Example:

FLUENT_PACKAGE_OPTIONS=""
# Control method to upgrade service (auto/manual) restart
FLUENT_PACKAGE_SERVICE_RESTART=auto

auto (default)

The service automatically restarts with zero-downtime restart feature when all of the following conditions are met:

1. The service was active before updating.
2. The installed and upgrading versions are required 5.2.0 or higher (both sides must support this feature).

The plugins will be automatically reinstalled if needed if your environment has an active online connection. For more details, please refer to "Automate Plugin Install for Update on Demand".

manual

You can use this approach if you prefer to manage plugins manually or if your environment does not have offline connection. If the service was active before the update, it will not restart automatically with this configuration. You need to restart the service manually.

• The zero-downtime restart:
  • Send a SIGUSR2 signal to the supervisor process, such as with kill -USR2 <PID>
• Normal restart:
  • Restart the service normally, such as with systemctl restart fluentd.

Automate Plugin Install for Update on Demand

Automate Plugin Management

If you prefer to manage plugins automatically, please set FLUENT_PACKAGE_SERVICE_RESTART to auto (default).

When the service restarts automatically using the zero-downtime restart feature, any missing plugins are automatically detected and reinstalled before the restart.

Previously, automatic restarts after updates were not recommended due to potential issues. If you manually install plugins, you need to reinstall them before restarting when the embedded Ruby version is updated.

To address this, this feature enables automatic plugin installation during the restart process. The steps are as follows:

1. Collect the current list of plugins before the update.
   • Recognize gems as plugins that has fluent-plugin- prefix.
2. After installing the updated package, detect missing plugins.
   • Compare the collected plugin list with the default plugins to identify any missing ones.
3. Install the missing plugins.

Manual Plugin Management

If you prefer to manage plugins manually, please set FLUENT_PACKAGE_SERVICE_RESTART to manual. This is useful in cases such as:

• Pinning specific plugin versions
• Operating in an offline environment

In this mode, automatic plugin installation and restarts are disabled. You can manually install plugins and send a SIGUSR2 signal to trigger the zero-downtime restart feature after the update.

Caution: if you use a custom unit file, need to migrate it to use this feature safely

If you use a custom unit file, such as /etc/systemd/system/fluentd.service, please remove these 2 lines.

Environment=GEM_HOME=/opt/fluentd/lib/ruby/gems/...
Environment=GEM_PATH=/opt/fluentd/lib/ruby/gems/...

We don't need GEM_HOME and GEM_PATH. They will be removed after v5.2.0, but if you use a custom unit file, you need to remove them manually. If these variables are set, the zero-downtime restart feature may not work correctly. It is because the new process inherits the original environment variables if using this feature.

Please refer to #713 for more details.

Download

Please see the download page.

About next LTS schedule

We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Read More

Fluentd v1.18.0 has been released

Hi users!

We have released v1.18.0 on 2024-11-29. ChangeLog is here.

This release is a new release of v1.18 series. In this release, we added zero-downtime-restart as a key feature, enhanced some features and fixed bugs.

Enhancement

Add zero-downtime-restart feature

In this release, we have added the feature to update the Fluentd process with zero downtime for Linux and macOS environments (not supported on Windows). You can use this feature to reload config and restart Fluentd gracefully.

You can use this feature by sending a SIGUSR2 signal to the supervisor process, and this changes behavior SIGUSR2 from graceful reload to zero-downtime-restart feature on Linux and macOS environments.

You can still use the conventional graceful reloading feature by directly sending SIGUSR2 to each worker process or using the RPC feature. However, it had some limitations and issues (Please see #4624 for details). So, we recommend using this new feature for graceful reloading/restarting unless there is a particular reason.

Please see #4624 for more information.

Add with-source-only feature

This feature has been added to support the zero-downtime-restart feature that it launches Fluentd with input plugins only.

This feature has added --with-source-only option in fluentd command, with_source_only option in system configuration and out_buffer plugin which can be used for buffering and relabeling events. These are used internally to support zero-downtime-restart.

Please see #4661 for more information.

Config File Syntax: Extend Embedded Ruby Code support for Hashes and Arrays

The Embedded Ruby Code has been supported in configuration. In this release, we have extended the syntax to allow the use of Embedded Ruby Code in Hashes and Arrays.

Example:

key1 ["foo","#{1 + 1}"] # Embedded Ruby Code in Array syntax
key2 {"foo":"#{1 + 1}"} # Embedded Ruby Code in Hash syntax

Result:

key1 ["foo","2"]
key2 {"foo":"2"}

Please note that this is not backward compatible, although we assume that this will never affect to actual existing configs.

In case the behavior changes unintentionally, you can disable this feature by surrounding the entire value with single quotes.

Example:

key '{"foo":"#{1 + 1}"}'

transport tls: Use SSL_VERIFY_NONE by default

We have changed the flags to SSL_VERIFY_NONE by default for server the certification verification.

In the previous versions, it had been necessary to set client_cert_auth false and insecure true to apply SSLVERIFYNONE. This was an inappropriate combination of settings. In this release, it can make it possible to apply SSL_VERIFY_NONE with just client_cert_auth false (default).

transport tls: Add ensure_fips option to ensure FIPS compliant mode

We have added ensure_fips option in transport section in config to check FIPS mode. When it configures ensure_fips true, it will check FIPS mode is supported in your environment, if not, just aborts. By default, it has disabled.

Example:

<transport tls>
  ensure_fips true
</transport>

Please see the document for more information.

plugin_helper/server: Add receive_buffer_size parameter in transport section

In the previous versions, the receive_buffer_size option can be set by in_udp only. In this release, we have added the receive_buffer_size option into transport section too to configure the buffer size on in_tcp or in_syslog plugins.

Example:

<transport udp>
  receive_buffer_size 4194304
</transport>

filter_parser: Now able to handle multiple parsed results

In the previous versions, there was a restriction that only the first record could be handled when the parsed result was multiple records. In this release, we have removed this limitation to handle multiple records appropriately.

You can read more about this limitation in the Remaining problem: filter_parser in Fluentd v1.17.0 release blog.

in_http: add add_tag_prefix option

We have added add_tag_prefix option in in_http plugin to add the prefix to the incoming event's tag, similar to in_forward plugin.

Example:

<source>
  @type http
  add_tag_prefix prod
</source>

System configuration: add path option in log section

We have added path option to log section of system configuration.

Example:

<system>
  <log>
    path /path/to/log_file
  </log>
</system>

Bug Fix

command: fix NoMethodError of --daemon under Windows

We have fixed a NoMethodError when launches fluentd with --daemon command line option under Windows platform.

fluentd command: fix --plugin (-p) option not to overwrite default value

We have fixed --plugin command line option that it does not overwrite the default value.

Enjoy logging!

Read More

fluent-package v5.0.5 has been released

Hi users!

We have released fluent-package v5.0.5. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.6 and Fluentd was updated to 1.16.6. We recommend upgrading to fluent-package v5.0.5!

Changes from fluent-package v5.0.4

In this release, focused on updating bundled ones.

• Update ruby to 3.2.6
• Update fluentd to 1.16.6
• Update bundled gems overall
• Solved possibility of DoS through "NoMemoryError"
• msi: set GEM_HOME/GEM_PATH in fluentd.bat
• Fixed SIGABORT error with fluent-plugin-systemd

Solved possibility of DoS through "NoMemoryError"

In the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.

When that message was sent, it might cause "NoMemoryError" on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.

Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled into fluent-package v5.0.5.

msi: set GEM_HOME/GEM_PATH in fluentd.bat

In the previous versions, fluentd.bat doesn't set GEMHOME/GEMPATH explicitly.

If users set custom GEM_HOME / GEM_PATH environment variables, the batch file will not work as expected. Therefore, this patch will set the appropriate values in the batch file.

Above bug was also fixed in fluent-package v5.0.5.

Fixed SIGABORT error with fluent-plugin-systemd

In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.

Download

Please see the download page.

About next LTS schedule

We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Read More

Scheduled support lifecycle announcement about Fluent Package v6

Hi users!

We had launched fluent-package v5 series last year, recently shipped v5.0.4 in LTS release channel and v5.1.0 in normal release channel.

In this blog article, we explains the planned next major updates - v6.0.0.

When the next LTS (Long Term Support) version available?

In short, we will ship fluent-package v6 in Aug, 2025.

We keeps two release channels as follows:

• Normal release
• LTS (Long Term Support)

Here is the difference of these channels.

• Normal release (6.x series)
  • Recommended for early adopters (eager to try new features of Fluentd)
  • Rapid release development (planned semi-annually releases, but it may vary.)
  • Fluentd will be upgraded occasionally
    • Minor upgrade will be applied. e.g. 1.18 => 1.19 and so on.
    • e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), ... v6.1.0 (Fluentd v1.19.0)
  • The only latest version will be supported (bug fix will not be backported into older version)
• LTS (Long Term Support, 6.0.x series)
  • Recommended for enterprise services
  • Security and bug fix release only
  • Fluentd will be upgraded only in the teeny version and will stick to 1.18.x series. (T.B.D.)
    • Thus, the version number will be 6.0.x. (6.1 series will not be shipped for LTS channel)
    • e.g. v6.0.1 (Fluentd v1.18.1), v6.0.2 (Fluentd v1.18.2), v6.0.3 (Fluentd v1.18.3) ... v6.0.x (Fluentd v1.18.x) and so on.
  • Next major version will be fluent-package v7.0.0, and will be shipped in 2027 (T.B.D.).

As for fluent-package v6,

• fluent-package v5 LTS will be supported until Dec, 2025.
  • We expect users upgrade from v5 to v6 during the grace period.
• Until 6.1.0 is released, normal release channel and LTS channel are the same package.
• Upgrade from v4 or older version is not supported. Upgrade to v5 first, then upgrade it to v6.

Current releases and future plans:

Happy logging!

Read More

Fluentd v1.16.6 has been released

Hi users!

We have released v1.16.6 on 2024-08-20. ChangeLog is here.

This release is a maintenance release of v1.16 series. In this release, some bugs were fixed with backporting from v1.17.x.

Bug Fixes

yaml_parser: $log_level notation in YAML was supported

In this release, we added a support for $log_level in YAML config file syntax.

In the previous versions, if you used log_level in YAML configuration, it causes a following warning:

[warn]: #0 'log_level' is deprecated parameter name. use '@log_level' instead.

But @log_level is invalid in YAML format. In YAML format, it should be $log_level as same as similar parameter such as $tag and $type, but not supported before.

Since Fluentd v1.16.6, $log_level has been supported! (In v1.17.x, it was supported since v1.17.1)

fluentd command: Fix --plugin (-p) option not to overwrite default value

In this release, we have fixed a bug that fluentd command: --plugin (-p) option could not overwrite default value.

This bug was unexpectedly introduced since v1.16.0.

Now it was reverted to the original behavior.

logger: Fix LoadError with console gem v1.25

In this release, we fixed a LoadError with console gem v1.25 or later.

Since console gem v1.25 or later, it's internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.

• 500 response when scraping prometheus metrics in kubernetes with istio service mesh

With supporting console gem v1.25 or later, this issue has solved!

Enjoy logging!

Read More