Fluentd Blog

Fluentd v1.19.0 has been released

Hi users!

We have released v1.19.0 on 2025-07-30. ChangeLog is here.

This release is a new release of v1.19 series. This release includes many new features and bug fixes, with a particular focus on reducing operational and running costs and improving overall stability.

$ rm Gemfile.lock
$ bundle

Major new features and improvements

Add feature to evacuate chunk files when retry limit

In this release, we added a buffer evacuation feature to the buf_file and buf_file_single plugins. This simplifies recovery after failure.

In previous versions, when an output plugin fails to deliver data by network issue, if the retry limit was exceeded and secondary output was not configured, the data would be discarded.

In this release, buffer files are automatically evacuated when the retry limit is exceeded. Once the issue is resolved, you can restore the evacuated files to the original buffer directory and restart Fluentd to resume sending the data.

You can even restart without causing downtime to input plugins like in_udp by using the zero-downtime restart feature.

Evacuated files are stored at:

${root_dir}/buffer/${plugin_id}/

Ref. Handling Successive Failures

Note: This feature is disabled when secondary outputs or retry_forever are used. To enable buffer evacuation, remove those settings.

Improved buffer corruption detection

Buffer files may become corrupted due to events like system crashes or sudden power down. In previous versions, Fluentd could fail to handle corrupted files properly, triggering errors on every restart. You had to manually identify and delete the corrupted files.

This release improves buffer corruption detection. When corruption is detected, the file is automatically moved to a backup directory, and startup errors are suppressed.

Enhanced metrics

We have improved Fluentd's observability by enhancing its metrics capabilities:

• Input plugin metrics are now enabled by default.
  • Previously, you needed to explicitly set enable_input_metrics to true. From this release, it's collected by default. To disable, set enable_input_metrics false.
• New metrics for in_tail:
  • tracked_file_count: Number of files currently being tailed.
• New metrics for output plugin:
  • write_secondary_count: Cumulative count of records written to secondary output.
  • drop_oldest_chunk_count: Cumulative count of chunks dropped when drop_oldest_chunk is used for overflow_action.

Performance improvement: switch from yajl-ruby to json gem

Previously, we used yajl-ruby for high-performance JSON processing, such as writing buffer chunks and parsing with parser_json.

However, in recent years, the standard Ruby json library has improved performance significantly. So we changed to use the json gem as the default JSON processor.

Zstandard (zstd) compression support

We have added support for the Zstandard (zstd) compression format, which offers both high compression ratios and faster processing compared to gzip.

• The following plugins now accept compress zstd:
  • buffer
  • out_file
  • out_forward

Note: To use zstd with out_forward, the destination server should support it. You cannot use it with Fluent Bit or Fluentd versions older than v1.19.0.

TLS 1.3 support

The out_http plugin now supports TLS 1.3, offering enhanced security. By default, TLS 1.2 is used. To enable TLS 1.3, set the tls_version parameter as follows:

<match tag>
  @type http
  ...
  tls_version TLSv1_3
  ...
</match>

Other useful improvements

• in_forward plugin:
  • skip_invalid_event is now enabled by default to skip corrupted records.
• out_stdout plugin:
  • Added the use_logger parameter. Setting use_logger false forces output to STDOUT regardless of Fluentd's own logging configuration. This is especially useful in container environments like Kubernetes.
• out_file plugin:
  • Added symlink_path_use_relative to allow relative paths in symlink_path.
• System configuration:
  • Added forced_stacktrace_level to control the log level of stack traces.

Major bug fixes

Fixed memory leak in formatter_csv plugin

To improve performance, it introduced a caching mechanism in formatter_csv, but incorrectly cached threads. This caused memory leaks when used with in_exec, which spawns a new thread per log entry.

This fix is backported to Fluentd v1.16.8.

in_tail plugin: fixed incorrect behavior with encoding option

The in_tail plugin provides encoding and from_encoding options.

• Expected usage:
  • Setting only encoding: Specifying encoding information.
  • Setting both encoding and from_encoding: Converting encoding.

Due to a bug since Fluentd v0.14.12 (at least from td-agent v3), when only encoding was set, unintended encoding conversion occurred, it might corrupt log data.

In previous versions, you need to specify the same encoding for both encoding and from_encoding as workaround.

Windows: fixed bug where service doesn't stop when the supervisor is dead

On Windows, if the supervisor process exited due to configuration errors, the service could continue running incorrectly. This has been fixed so the service now stops as expected, improving error detection.

This fix is backported to Fluentd v1.16.8.

Fixed --umask Bug

Fixed a bug where the --umask setting was ignored when used together with --daemon, leading to unexpected file permissions.

Enjoy logging!

Read More

fluent-package v5.0.7 has been released

Hi users!

We have released fluent-package v5.0.7. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)

This is a maintenance release of v5.0.x LTS series. Bundled Fluentd was updated to 1.16.9.

We recommend upgrading to fluent-package v5.0.7!

Changes from fluent-package v5.0.6

• Update bundled ruby to 3.2.8
• Update bundled Fluentd to v1.16.9
  • Fluentd v1.16.8 has been released
  • Fluentd v1.16.9 has been released
• Update bundled gems overall
• Update bundled openssl gem to 3.3.0 due to support FIPS
• Fix memory leaks when exception was raised frequently with fluent-plugin-elasticsearch

Update bundled openssl gem to 3.3.0 due to support FIPS

With an old gem, it caused an exception after FIPS enabled in OpenSSL. We have updated the bundled openssl gem to 3.3.0 to solve the issue.

Fix memory leaks when exception was raised frequently with fluent-plugin-elasticsearch

There were memory leaks under conditions where exceptions frequently occurred during communication between Elasticsearch and fluent-plugin-elasticsearch. The issue was fixed in fluent-plugin-elasticsearch v5.4.4 and bundled into fluent-package v5.0.7.

Download

Please see the download page.

Announcement

About next LTS schedule

We plan to ship the next LTS version of fluent-package v5.0.8 on September, 2025. The content of updates are still in T.B.D.

End of support for td-agent v4, let's migrate to fluent-package

As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 reached EOL in Dec, 2023.

After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.

We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5

Follow us on X

We have been posting information about Fluentd in Japanese on @fluentd_jp. We would appreciate it if you followed the X account.

Read More

Fluentd v1.16.9 has been released

Hi users!

We have released v1.16.9 on 2025-05-14. ChangeLog is here.

This release is a maintenance release of v1.16 series. In this release, we have fixed a critical bug of the previous version, v1.16.8. Please do not use v1.16.8.

This release will be bundled for upcoming fluent-package LTS version v5.0.7!

Bug Fixes

Windows: Fix bug where the service accidentally stops after starting

In the previous version v1.16.8, the Windows service (fluentdwinsvc) accidentally stops after starting without stopping the supervisor and worker processes. This is a critical bug of v1.16.8 for users who use Fluentd as a Windows service.

In this release, we have fixed this bug.

See #4954 for more information.

Enjoy logging!

Read More

Fluentd v1.16.8 has been released

Hi users!

We have released v1.16.8 on 2025-05-01. ChangeLog is here.

This release is a maintenance release of v1.16 series.

This release will be bundled for upcoming fluent-package LTS version v5.0.7!

Bug Fixes

Windows: Stop the service correctly when the supervisor is dead

In this release, we have fixed a bug that Windows service process will continue running even if the supervisor process is dead due to config error or etc.

With fluentd#4942 fixed, it becomes easier to notice config errors and other issues.

formatter_csv: Fixed memory leak

The caching mechanism introduced for performance optimization was caching threads inappropriately in formatter_csv. The in_exec plugin creates a thread each time to emit an event. Then all threads were cached in formatter_csv, causing a memory leak.

With fluentd#4920 fixed, it becomes stable.

Misc news

Windows: Add fiddle as dependency gem for Ruby 3.5

For comming Ruby 3.5, we have added a fiddle gem as dependency.

CI stabilization

We are enforcing stabilizing CI of Fluentd. It has nothing to do with Fluentd's functionality directly, but it is very important to provide stable version of Fluentd continuously.

Enjoy logging!

Read More

Drop Alpine and switch default to Debian for Fluentd docker image

Hi users!

We have planned to change what docker image provides.

Currently, we ship Fluentd docker image for Alpine and Debian (armhf, arm64, amd64). But, Alpine image in Fluentd was already marked as deprecated, so we will make things forward.

In the near future, we will drop Alpine image. This decision affects existing Alpine image users.

Notice about existing Alpine image users

• fluent/fluentd:edge will be changed from Alpine to Debian image
  • This is a notable incompatible change!
• v1.19 (not released yet)
  • No more releases for Alpine images
  • Then short tag also changed to Debian images (e.g. v1.19.0-1.0, v1.19-1 should be Debian instead of Alpine)
• v1.18 Alpine image will not be supported after v1.19 is released (T.B.D.), so recommend to migrate it!
• v1.16 Alpine image will be supported until Dec 2025, so there is room to migrate gradually.

If you still want to use Alpine, keep v1.16 series or v1.18 series. (not recommended though)

Why not provide Alpine anymore?

It was well known that Alpine has possibility of incompatibility and performance issues in contrast to Debian images. It was a historical reason to provide Alpine images - "In the previous versions, we provided them", so there is no positive reason to continue it.

Provide shorter tags to follow internal version bump

Not only just dropping Alpine image, we will improve tagging rules:

• Add version-specific tag which will not be affected by internal version bump

In previous versions, there was no version-specific tag which would follow internal version bump. For example, if you want to stick to Fluentd v1.18.0, you can select v1.18.0-1.0, but if internal version was bumped to v1.18.0-1.1 or something, need to update it. There is edge-debian tag, but it does not stick to a specific Fluentd version because it will automatically update when the next major/minor version is released.

For such a purpose, shorter v1.19 or v1.19.x tag will be available in the future release.

Happy logging!

Read More