This document describes the motivation and design of running Receiver in a stateless mode that does not have capabilities to store samples, it only routes remote write to further receivers based on hashring.
This allows setting optional deployment model were only routing receivers are using hashring files and does the routing and replication. That allows ingesting receivers to not handle any routing or hashring, only receiving multi tenant writes.
Currently, any change to the hashring configuration file will trigger all Thanos Receive nodes to flush their multi-TSDBs, causing them to enter an unready state until the flush is complete. This unavailability during a flush allows for a clear state transition, however it can result in downtimes on the order of five minutes for every configuration change. Moreover, during configuration changes, the hashring goes through an even longer period of partial unreadiness, where some nodes begin and finish flushing before and after others. During this partial unreadiness, the hashring can expect high internal request failure rates, which cause clients to retry their requests, resulting in even higher load. Therefore, when the hashring configuration is changed due to automatic horizontal scaling of a set of Thanos Receivers, the system can expect higher than normal resource utilization, which can create a positive feedback loop that continuously scales the hashring.
We propose allowing to run Thanos Receiver in a mode that only forwards/replicates remote write (distributor mode) apart from the ingesting mode (default mode). You can enable that mode by simply not specifying:
Endpoint of local receive node. Used to
identify the local node in the hashring
We can call this mode a “Routing Receiver”. Similarly, we can skip specifying any hashring to Thanos Receiver (
--receive.hashrings-file=<path>), explicitly purposing it only for ingesting. We can call this mode “Ingesting Receiver”.
User can also mix all of those two modes for various federated hashrings etc. So instead of what we had before:
This allows us to (optionally) model deployment in a way that avoid expensive re-configuration of the stateful ingesting Receivers after the hashring configuration file has changed.
In comparison to previous proposal (as mentioned in alternatives we have big advantages:
--receive.hashrings does not forward or replicate requests, it routes straight to multi-tsdb.
--Receiver.local-endpoint will assume that no storage is needed, so will skip creating any resources for multi TSDB.
This potentially makes the receiver more difficult to operate and understandable for Thanos users. I would argue this is however much harder in overall Thanos deployment. Otherwise, this option is exactly the same.
Idea would be similar same as in Proposal, but there will be explicit flag to turn off local storage capabilities.
I think we can have much more understandable logic if we simply not configure hashring for ingesting receivers and not configure local hashring endpoint to notify that such Receiver instance will never store anything.