Hi users!
We have released fluent-package v5.2.0. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This release is a new release of v5.2 series.
GEM_HOME/GEM_PATH in fluentd.batPreviously, when upgrading the fluent-package, you had to stop the fluentd service, install the new version of the fluent-package, then install any necessary plugins, and finally restart the fluentd service.
Starting from fluent-package v5.2.0, we have introduced the zero-downtime restart feature that automatically installs the required plugins during the upgrade process and allows you to upgrade Fluentd without interrupting the logging service.
Please refer to "Advisory for upgrading with zero-downtime".
In the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.
When that message was sent, it might cause "NoMemoryError" on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.
Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled.
GEM_HOME/GEM_PATH in fluentd.batIn the previous versions, fluentd.bat doesn't set GEMHOME/GEMPATH explicitly.
If users set custom GEM_HOME / GEM_PATH environment variables, the
batch file will not work as expected. Therefore, this patch will set
the appropriate values in the batch file.
We have changed so that Fluentd does not restart when the needrestart package is installed.
fluent-package v5.2.0 places /etc/needrestart/conf.d/50-fluent-package.conf.
In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.
The zero-downtime restart feature can be configured by FLUENT_PACKAGE_SERVICE_RESTART environment variable.
Please refer to following section for more details.
FLUENT_PACKAGE_SERVICE_RESTART environmental variable
FLUENT_PACKAGE_SERVICE_RESTART environment variable.FLUENT_PACKAGE_SERVICE_RESTART environmental variableWe have introduced FLUENT_PACKAGE_SERVICE_RESTART environment variable to configure the zero-downtime restart feature.
The variable exists in the following file:
/etc/sysconfig/fluentd/etc/default/fluentdExample:
FLUENT_PACKAGE_OPTIONS=""
# Control method to upgrade service (auto/manual) restart
FLUENT_PACKAGE_SERVICE_RESTART=auto
The service automatically restarts with zero-downtime restart feature when all of the following conditions are met:
The plugins will be automatically reinstalled if needed if your environment has an active online connection. For more details, please refer to "Automate Plugin Install for Update on Demand".
You can use this approach if you prefer to manage plugins manually or if your environment does not have offline connection. If the service was active before the update, it will not restart automatically with this configuration. You need to restart the service manually.
SIGUSR2 signal to the supervisor process, such as with kill -USR2 <PID>systemctl restart fluentd.If you prefer to manage plugins automatically, please set FLUENT_PACKAGE_SERVICE_RESTART to auto (default).
When the service restarts automatically using the zero-downtime restart feature, any missing plugins are automatically detected and reinstalled before the restart.
Previously, automatic restarts after updates were not recommended due to potential issues. If you manually install plugins, you need to reinstall them before restarting when the embedded Ruby version is updated.
To address this, this feature enables automatic plugin installation during the restart process. The steps are as follows:
fluent-plugin- prefix.If you prefer to manage plugins manually, please set FLUENT_PACKAGE_SERVICE_RESTART to manual.
This is useful in cases such as:
In this mode, automatic plugin installation and restarts are disabled.
You can manually install plugins and send a SIGUSR2 signal to trigger the zero-downtime restart feature after the update.
If you use a custom unit file, such as /etc/systemd/system/fluentd.service, please remove these 2 lines.
Environment=GEM_HOME=/opt/fluentd/lib/ruby/gems/...
Environment=GEM_PATH=/opt/fluentd/lib/ruby/gems/...
We don't need GEM_HOME and GEM_PATH.
They will be removed after v5.2.0, but if you use a custom unit file, you need to remove them manually.
If these variables are set, the zero-downtime restart feature may not work correctly.
It is because the new process inherits the original environment variables if using this feature.
Please refer to #713 for more details.
Please see the download page.
We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Hi users!
We have released v1.18.0 on 2024-11-29. ChangeLog is here.
This release is a new release of v1.18 series. In this release, we added zero-downtime-restart as a key feature, enhanced some features and fixed bugs.
In this release, we have added the feature to update the Fluentd process with zero downtime for Linux and macOS environments (not supported on Windows). You can use this feature to reload config and restart Fluentd gracefully.
You can use this feature by sending a SIGUSR2 signal to the supervisor process,
and this changes behavior SIGUSR2 from graceful reload to zero-downtime-restart feature on Linux and macOS environments.
You can still use the conventional graceful reloading feature by directly sending SIGUSR2 to each worker process or using the RPC feature.
However, it had some limitations and issues (Please see #4624 for details).
So, we recommend using this new feature for graceful reloading/restarting unless there is a particular reason.
Please see #4624 for more information.
This feature has been added to support the zero-downtime-restart feature that it launches Fluentd with input plugins only.
This feature has added --with-source-only option in fluentd command, with_source_only option in system configuration and out_buffer plugin which can be used for buffering and relabeling events.
These are used internally to support zero-downtime-restart.
Please see #4661 for more information.
The Embedded Ruby Code has been supported in configuration. In this release, we have extended the syntax to allow the use of Embedded Ruby Code in Hashes and Arrays.
Example:
key1 ["foo","#{1 + 1}"] # Embedded Ruby Code in Array syntax
key2 {"foo":"#{1 + 1}"} # Embedded Ruby Code in Hash syntax
Result:
key1 ["foo","2"]
key2 {"foo":"2"}
Please note that this is not backward compatible, although we assume that this will never affect to actual existing configs.
In case the behavior changes unintentionally, you can disable this feature by surrounding the entire value with single quotes.
Example:
key '{"foo":"#{1 + 1}"}'
SSL_VERIFY_NONE by defaultWe have changed the flags to SSL_VERIFY_NONE by default for server the certification verification.
In the previous versions, it had been necessary to set client_cert_auth false and insecure true to apply SSLVERIFYNONE.
This was an inappropriate combination of settings.
In this release, it can make it possible to apply SSL_VERIFY_NONE with just client_cert_auth false (default).
ensure_fips option to ensure FIPS compliant modeWe have added ensure_fips option in transport section in config to check FIPS mode.
When it configures ensure_fips true, it will check FIPS mode is supported in your environment, if not, just aborts.
By default, it has disabled.
Example:
<transport tls>
ensure_fips true
</transport>
Please see the document for more information.
receive_buffer_size parameter in transport sectionIn the previous versions, the receive_buffer_size option can be set by in_udp only.
In this release, we have added the receive_buffer_size option into transport section too to configure the buffer size on in_tcp or in_syslog plugins.
Example:
<transport udp>
receive_buffer_size 4194304
</transport>
In the previous versions, there was a restriction that only the first record could be handled when the parsed result was multiple records. In this release, we have removed this limitation to handle multiple records appropriately.
You can read more about this limitation in the Remaining problem: filter_parser in Fluentd v1.17.0 release blog.
add_tag_prefix optionWe have added add_tag_prefix option in in_http plugin to add the prefix to the incoming event's tag, similar to in_forward plugin.
Example:
<source>
@type http
add_tag_prefix prod
</source>
We have added path option to log section of system configuration.
Example:
<system>
<log>
path /path/to/log_file
</log>
</system>
We have fixed a NoMethodError when launches fluentd with --daemon command line option under Windows platform.
We have fixed --plugin command line option that it does not overwrite the default value.
Enjoy logging!
Hi users!
We have released fluent-package v5.0.5. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.6 and Fluentd was updated to 1.16.6. We recommend upgrading to fluent-package v5.0.5!
In this release, focused on updating bundled ones.
GEM_HOME/GEM_PATH in fluentd.batIn the previous versions of msgpack gem, there was a possibility that cause DoS (Denial Of Service) when crafted message was sent to Fluentd.
When that message was sent, it might cause "NoMemoryError" on Fluentd because there is a case that msgpack try to pre-allocate huge amount of memories.
Above behavior was fixed not to pre-allocate over 32k entries (limit pre-allocated amount of memories) in msgpack 1.7.3 and bundled into fluent-package v5.0.5.
GEM_HOME/GEM_PATH in fluentd.batIn the previous versions, fluentd.bat doesn't set GEMHOME/GEMPATH explicitly.
If users set custom GEM_HOME / GEM_PATH environment variables, the
batch file will not work as expected. Therefore, this patch will set
the appropriate values in the batch file.
Above bug was also fixed in fluent-package v5.0.5.
In this release, the bundled fluent-plugin-systemd was updated to 1.1.0. That release fixed SIGABORT error with inconsistency of memory allocator handling.
Please see the download page.
We plan to ship the next LTS version of fluent-package v5.0.6 on March, 2025. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Hi users!
We had launched fluent-package v5 series last year, recently shipped v5.0.4 in LTS release channel and
v5.1.0 in normal release channel.
In this blog article, we explains the planned next major updates - v6.0.0.
In short, we will ship fluent-package v6 in Aug, 2025.
We keeps two release channels as follows:
Here is the difference of these channels.
fluent-package v7.0.0, and will be shipped in 2027 (T.B.D.).As for fluent-package v6,
Current releases and future plans:
| Release Version | Release Date | End of life |
|---|---|---|
| v5.0.x (LTS) | Aug, 2023 | Dec, 2025 |
| v6.0.0 (LTS) | Aug, 2025 | Dec, 2027 (T.B.D.) |
| v7.0.0 (LTS) | Aug, 2027 (T.B.D.) | T.B.D. |
Happy logging!
Hi users!
We have released v1.16.6 on 2024-08-20. ChangeLog is here.
This release is a maintenance release of v1.16 series. In this release, some bugs were fixed with backporting from v1.17.x.
yaml_parser: $log_level notation in YAML was supportedIn this release, we added a support for $log_level in YAML config file syntax.
In the previous versions, if you used log_level in YAML configuration, it causes a following warning:
[warn]: #0 'log_level' is deprecated parameter name. use '@log_level' instead.
But @log_level is invalid in YAML format. In YAML format, it should be $log_level as same as
similar parameter such as $tag and $type, but not supported before.
Since Fluentd v1.16.6, $log_level has been supported! (In v1.17.x, it was supported since v1.17.1)
--plugin (-p) option not to overwrite default valueIn this release, we have fixed a bug that fluentd command: --plugin (-p) option could not overwrite default value.
This bug was unexpectedly introduced since v1.16.0.
Now it was reverted to the original behavior.
In this release, we fixed a LoadError with console gem v1.25 or later.
Since console gem v1.25 or later, it's internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.
With supporting console gem v1.25 or later, this issue has solved!
Enjoy logging!
Hi users!
We have released v1.17.1 on 2024-08-19. ChangeLog is here.
This release is a new release of v1.17 series. In this release, we added some new options for some plugins and fixed bugs of Parser.
yaml_parser: $log_level notation in YAML was supportedIn this release, we added a support for $log_level in YAML config file syntax.
In the previous versions, if you used log_level in YAML configuration, it causes a following warning:
[warn]: #0 'log_level' is deprecated parameter name. use '@log_level' instead.
But @log_level is invalid in YAML format. In YAML format, it should be $log_level as same as
similar parameter such as $tag and $type, but not supported before.
Since Fluentd v1.17.1, $log_level has been supported now!
out_http: compress option was supportedIn this release, we added a new option compress for the out_http plugin.
You can compress HTTP request body by specifying compress gzip.
To keep compatibility, the default configuration is set as compress text (no compression).
This feature was contributed by @rockliffelewis. Thanks!
in_sample: the default behavior was changedIn this release, we changed the default behavior of in_sample plugin and added a new option reuse_record for it.
The default behavior has changed to copy sample data to avoid the impact of destructive changes by subsequent plugins.
In the previous versions, when in_sample plugin is used with the subsequent filter parser plugin which uses remove_key_name_field, it raises the following error repeatedly.
#0 dump an error event: error_class=ArgumentError error="message does not exist"
This error occurs when key_name and remove_key_name_field removes key from record with destructive change in filter parser.
It affects generated sample data after twice or later.
(This error is not limited to filter parser plugin. The similar destructive change will be done with the subsequent plugins, this error is reproducible)
The new default behavior (reuse_record false) avoids this problem, but it increases the load when generating large amounts of sample data.
You can use this new parameter to have the same performance as before with reuse_record.
In this release, we fixed a LoadError with console gem v1.25 or later.
Since console gem v1.25 or later, it's internal change causes a LoadError in Fluentd. This error breaks existing plugin such as fluent-plugin-prometheus which provides monitoring REST API.
With supporting console gem v1.25 or later, this issue has solved!
Enjoy logging!
Hi users!
We have released fluent-package v5.1.0. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This release is a new release of v5.1 series. Bundled ruby version was upgraded to 3.2.5 and fluentd v1.17.0 was shipped!
In this release, focused on updating bundled ones.
In LTS version, conservative updating policy is enabled, so updating gem is very limited to bug fix or security fix. In contrast to LTS version, v5.1.0 bundles more recent components because of standard version.
As we already announced in Scheduled support lifecycle announcement about Fluent Package blog article, in normal release channel, we will ship the latest version (v1.17.0) of Fluentd.
So if you want to try using latest version of Fluentd, v5.1.0 is one for you. If you use fluent-package in enterprise services, keep using fluent-package 5.0.x (LTS).
Please see the download page.
We plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Hi users!
We have released fluent-package v5.0.4. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This is a maintenance release of v5.0.x LTS series. As bundled Ruby was updated to 3.2.4 and a foolproof mechanism was implemented to prevent launching duplicated Fluentd instances, we recommend upgrading to fluent-package v5.0.4!
In this release, a foolproof mechanism was implemented to prevent launching Fluentd wrongly if the service is already running.
As you know, you can check the version of Fluentd with fluentd --version, but there is a case that fluentd -v is executed wrongly to
do it.
When already running Fluentd as a service, fluentd -v launches a duplicated Fluentd instance with the same fluentd configuration.
If you launch duplicated Fluentd instances, it causes the corruption of processing Fluentd buffer.
To prevent such a situation, a foolproof was implemented now.
For example, if Fluentd is running as a service, launching Fluentd causes an error to block it.
Here is the example on Windows:
> fluentd
Error: Can't start duplicate Fluentd instance with the default config.
To start Fluentd, please do one of the following:
(Caution: Please be careful not to start multiple instances with the same config.)
- Stop the Fluentd Windows service 'fluentdwinsvc'.
- Specify the config path explicitly by '-c' ('--config').
Even though if you wrongly launch Fluentd to check version with -v (It should be --version to show version), then it causes the following error.
> fluentd -v
Error: Can't start duplicate Fluentd instance with the default config.
To take the version, please use '--version', not '-v' ('--verbose').
To start Fluentd, please do one of the following:
(Caution: Please be careful not to start multiple instances with the same config.)
- Stop the Fluentd Windows service 'fluentdwinsvc'.
- Specify the config path explicitly by '-c' ('--config')
Note that this foolproof feature is intended to block launching duplicated Fluentd instance, you can explicitly launch Fluentd by specifying a specific option to pass it even though already Fluentd is running as a service.
On Windows:
-c (--config)--dry-run--reg-winsvc--reg-winsvc-fluentdopt--show-plugin-configOn Linux:
-c (--config)--dry-run--show-plugin-configWe plan to ship the next LTS version of fluent-package v5.0.5 on Oct, 2024. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Please see the download page.
Hi users!
We have released v1.17.0 on 2024-04-30. ChangeLog is here.
This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.
in_tail: Add glob_policy option for expanding glob capability of path and exclude_pathIn this release, we added a new option glob_policy for in_tail plugin.
In previous versions, we can use only * in glob patterns for path and exclude_path option.
Example:
path /path/to/*
exclude_path ["/path/to/*.gz", "/path/to/*.zip"]
From this version, we can also use [], ?, and {} in glob patterns depending on the glob_policy option.
Example:
path "[0-1][2-3].log"
glob_policy extended
Please see the document and #4401 for more information.
out_http: Support AWS Signature Version 4 authenticationIn this release, we added a new option aws_sigv4 for the method setting of out_http plugin.
By using this option, out_http can use AWS Signature Version 4.
For example, this allows out_http to write to Amazon OpenSearch Ingestion.
Please see the document and #4459 for more information.
out_http: Add option to reuse connectionsIn this release, we add a new option reuse_connections for out_http plugin.
This option will improve throughput of out_http.
Please see the document and #4330 for more information.
in_http: Recognize CSP reports as JSON dataIn this release, we make the data type of the request where the Content-Type is application/csp-report be considered JSON by default.
Now, in_http can receive Content Security Policy's report by default.
Please see #4282 for more information.
The record data in an event of Fluentd must be a hash object.
However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.
In this release, the following parser plugins have been fixed.
The changes are as follows:
Here are the details for each case.
Config:
<source>
@type tcp
tag test.tcp
<parse>
@type json
</parse>
</source>
<match test.**>
@type stdout
</match>
Send an array data:
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
The result before this version:
{datetime} test.tcp: [{"k":"v"},{"k2":"v2"}]
The result after this version:
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
Config:
<source>
@type tcp
tag test.tcp
<parse>
@type json
null_empty_string
</parse>
</source>
<match test.**>
@type stdout
</match>
Send an array data:
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
The result before this version:
{datetime} [error]: #0 unexpected error on reading data host="xxx" port=xxx error_class=NoMethodError error="undefined method `each_key' for [{\"k\"=>\"v\"}, {\"k2\"=>\"v2\"}]:Array"
The result after this version:
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
In the previous versions, filter_parser could return an array record based on this wrong behavior.
From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case.
This behavior should improve in the future.
Here is an example.
<source>
@type sample
tag test.array
sample {"message": "[{\"k\":\"v\"}, {\"k2\":\"v2\"}]"}
</source>
<filter test.**>
@type parser
key_name message
<parse>
@type json
</parse>
</filter>
<match test.**>
@type stdout
</match>
The result before this version:
{datetime} test.array: [{"k":"v"},{"k2":"v2"}]
The result after this version:
{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: '[{\"k\":\"v\"}, {\"k2\":\"v2\"}]'" location=nil tag="test.array" time=xxx record={"k2"=>"v2"}
{datetime} test.array: {"k":"v"}
These are the major changes for this release.
In addition, some performance improvements have been included! Please see ChangeLog for details!
Enjoy logging!
Hi users!
We have released fluent-package v5.0.3. fluent-package is a stable distribution package of Fluentd. (successor of td-agent v4)
This is a maintenance release of v5.0.x LTS series. As significant slow starting service and crash issues during startup on Windows were fixed, we recommend upgrading to fluent-package v5.0.3!
%~dp0 correctly.We plan to ship next LTS version of fluent-package v5.0.4 on June, 2024. The content of updates are still in T.B.D.
As it was already announced Drop schedule announcement about EOL of Treasure Agent (td-agent) 4, td-agent v4 was reached EOL in Dec, 2023.
After reached EOL, td-agent v4.5.3 on Windows was released because there was a crash bug during startup on Windows. It was backported fix from fluent-package v5 as it is critical in some case. Even though this was a exceptional maintenance release, but there is no change to the fact that we already stopped maintaining td-agent v4.
We strongly recommend migrating from td-agent v4 to fluent-package v5 (LTS). See Upgrade to fluent-package v5
Please see the download page.
Fluentd is an open source data collector to simplify log management.
2024-08-29: Scheduled support lifecycle announcement about Fluent Package v6
2023-08-29: Drop schedule announcement about EOL of Treasure Agent (td-agent) 4
2023-08-29: Scheduled support lifecycle announcement about Fluent Package
2023-07-31: Upgrade to fluent-package v5
2024-12-14: fluent-package v5.2.0 has been released
2024-11-29: Fluentd v1.18.0 has been released
2024-11-08: fluent-package v5.0.5 has been released
2024-08-29: Scheduled support lifecycle announcement about Fluent Package v6
2024-08-20: Fluentd v1.16.6 has been released
2024-08-19: Fluentd v1.17.1 has been released
2024-08-02: fluent-package v5.1.0 has been released
2024-07-02: fluent-package v5.0.4 has been released
2024-04-30: Fluentd v1.17.0 has been released
2024-03-29: fluent-package v5.0.3 has been released
Want to learn the basics of Fluentd? Check out these pages.
Couldn't find enough information? Let's ask the community!
You need commercial-grade support from Fluentd committers and experts?
©2010-2024 Fluentd Project. ALL Rights Reserved.
Fluentd is a hosted project under the Cloud Native Computing Foundation (CNCF). All components are available under the Apache 2 License.