Telegraf Google Cloud PubSub 输入插件实战订阅拉取、消息解码与可靠投递原理【免费下载链接】telegrafAgent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.项目地址: https://gitcode.com/GitHub_Trending/te/telegraf本文以 Telegraf 仓库中 cloud_pubsub 插件文档 为核心系统讲解如何通过inputs.cloud_pubsub从 Google Cloud PubSub 拉取消息并解析为指标数据。读完本文你将掌握该插件的完整配置参数、服务型输入的运行方式、基于 tracking metrics 的可靠确认Ack机制以及消息长度限制、gzip 解压、Base64 解码等消息预处理链路并可从源码与测试层面理解其底层实现。插件概览与适用场景cloud_pubsub是 Telegraf 的一个服务型输入插件Service Input自Telegraf v1.10.0起提供属于cloud, messaging类别支持所有平台 all。它从 Google Cloud PubSub 服务消费消息并使用 Telegraf 支持的各种 输入数据格式 将消息解析为指标。典型的应用场景包括将 GCP 上各类服务如云日志导出、审计日志、事件流投递到 PubSub 的消息实时接入 Telegraf 的指标管道。说明pubsub与data_formats均指 Google Cloud 与 Telegraf 官方文档本文以仓库内 输入数据格式说明 为解析格式参考。服务型输入与普通输入的关键差异与普通输入插件按interval定时采集不同服务型插件会启动一个常驻服务去监听并等待指标或事件到达。根据仓库中 服务型输入说明两者有两个关键差异全局或插件级的interval设置对服务型插件可能不生效CLI 选项--test、--test-wait、--once对该插件可能不会产生输出。这一点在源码中也有印证cloud_pubsub的Gather()方法为空实现return nil真正的数据流完全由Start()启动的接收协程驱动而非周期性的Gather采集。Tracking Metrics消息确认与不丢数据的基石该插件支持 tracking metrics即 Telegraf 在指标被所有输出成功写入后才通知插件插件随即向 PubSub 发送 Ack从而保证消息至少送达一次的可靠性语义。在 cloud_pubsub.go 中Start()首先执行ps.sem make(semaphore, ps.MaxUndeliveredMessages) ps.acc ac.WithTracking(ps.MaxUndeliveredMessages)插件内部维护一张undelivered映射表map[telegraf.TrackingID]message记录已从 PubSub 读取但尚未被任何输出写入的消息。整个确认闭环如下收到消息并成功解析出指标后调用ps.acc.AddTrackingMetricGroup(metrics)取得 TrackingID将原始 PubSub 消息存入undelivered表后台协程waitForDelivery()监听ps.acc.Delivered()通道每当一个指标组被全部输出确认送达就从表中取出对应消息并调用msg.Ack()见 waitForDelivery 实现若指标组写入失败消息不会被 AckPubSub 将在 ACK 截止时间后重新投递保证数据不因输出故障而丢失。max_undelivered_messages通过信号量semchannel控制已从 broker 读取但未写入输出的最大消息数防止内存无限增长。该值需要与代理的metric_batch_size统筹考虑设置过大可能造成源源不断的批次涌向输出设置过小则可能永远无法清空 broker 中的积压消息。其默认值1000在 源码常量定义 中可见。完整配置与参数逐项解析插件配置的权威来源是 sample.confREADME中的sample.conf即由此生成。以下为完整可用配置# Read metrics from Google PubSub [[inputs.cloud_pubsub]] ## Required. Name of Google Cloud Platform (GCP) Project that owns ## the given PubSub subscription. project my-project ## Required. Name of PubSub subscription to ingest metrics from. subscription my-subscription ## Required. Data format to consume. ## Each data format has its own unique set of configuration options. ## Read more about them here: ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md data_format influx ## Optional. Filepath for GCP credentials JSON file to authorize calls to ## PubSub APIs. If not set explicitly, Telegraf will attempt to use ## Application Default Credentials, which is preferred. # credentials_file path/to/my/creds.json ## Optional. Number of seconds to wait before attempting to restart the ## PubSub subscription receiver after an unexpected error. ## If the streaming pull for a PubSub Subscription fails (receiver), ## the agent attempts to restart receiving messages after this many seconds. # retry_delay_seconds 5 ## Optional. Maximum byte length of a message to consume. ## Larger messages are dropped with an error. If less than 0 or unspecified, ## treated as no limit. # max_message_len 1000000 ## Max undelivered messages ## This plugin uses tracking metrics, which ensure messages are read to ## outputs before acknowledging them to the original broker to ensure data ## is not lost. This option sets the maximum messages to read from the ## broker that have not been written by an output. ## ## This value needs to be picked with awareness of the agents ## metric_batch_size value as well. Setting max undelivered messages too high ## can result in a constant stream of data batches to the output. While ## setting it too low may never flush the brokers messages. # max_undelivered_messages 1000 ## The following are optional Subscription ReceiveSettings in PubSub. ## Read more about these values: ## https://godoc.org/cloud.google.com/go/pubsub/v2#ReceiveSettings ## Optional. Maximum number of seconds for which a PubSub subscription ## should auto-extend the PubSub ACK deadline for each message. If less than ## 0, auto-extension is disabled. # max_extension 0 ## Optional. Maximum number of unprocessed messages in PubSub ## (unacknowledged but not yet expired in PubSub). ## A value of 0 is treated as the default PubSub value. ## Negative values will be treated as unlimited. # max_outstanding_messages 0 ## Optional. Maximum size in bytes of unprocessed messages in PubSub ## (unacknowledged but not yet expired in PubSub). ## A value of 0 is treated as the default PubSub value. ## Negative values will be treated as unlimited. # max_outstanding_bytes 0 ## Optional. Max number of goroutines a PubSub Subscription receiver can spawn ## to pull messages from PubSub concurrently. This limit applies to each ## subscription separately and is treated as the PubSub default if less than ## 1. Note this setting does not limit the number of messages that can be ## processed concurrently (use max_outstanding_messages instead). # max_receiver_go_routines 0 ## Optional. If true, Telegraf will attempt to base64 decode the ## PubSub message data before parsing. Many GCP services that ## output JSON to Google PubSub base64-encode the JSON payload. # base64_data false ## Content encoding for message payloads, can be set to gzip or ## identity to apply no encoding. # content_encoding identity ## If content encoding is not identity, sets the maximum allowed size, ## in bytes, for a message payload when its decompressed. Can be increased ## for larger payloads or reduced to protect against decompression bombs. ## Acceptable units are B, KiB, KB, MiB, MB... # max_decompression_size 500MB必填参数project拥有该 PubSub 订阅的 GCP 项目名。源码 Init() 校验 中project与subscription任一为空都会直接返回project is required/subscription is required错误。subscription要消费消息的 PubSub 订阅名。注意是订阅而非主题——插件假设你已预先为主题创建好 Pull 订阅。data_format消息的解析格式默认示例为influx可配置 Telegraf 支持的任何 输入数据格式每种格式有各自的专属配置项。凭据配置credentials_file指定 GCP 凭据 JSON 文件的路径。若未设置Telegraf 将尝试使用Application Default Credentials (ADC)这也是官方推荐的方式。从 getPubSubClient 实现 可以看到两层逻辑配置了credentials_file时先通过common_gcp.ParseCredentialType解析凭据类型再以option.WithAuthCredentialsFile构建客户端选项未配置时调用google.FindDefaultCredentials(..., pubsub.ScopeCloudPlatform)查找 ADC查找失败则报错提示要么设置 ADC要么提供 CredentialsFile 配置。此外客户端还会附加internal.ProductToken()作为 User-Agent并显式声明pubsub.ScopeCloudPlatform权限范围。接收行为参数对应 PubSub ReceiveSettings以下参数直接映射到 Go 客户端pubsub.ReceiveSettings见 getGCPSubscription 实现max_extension每条消息自动延长 PubSub ACK 截止时间的最大秒数小于 0 时禁用自动延长。默认0。max_outstanding_messagesPubSub 中未处理消息未确认且未过期的最大条数0采用 PubSub 默认值负值视为无限制。max_outstanding_bytes同上但以字节为单位限制未处理消息总量。max_receiver_go_routines订阅接收器可并发拉取消息的 goroutine 上限该限制按订阅独立生效小于 1 时采用 PubSub 默认值。注意它不限制消息的并发处理数量——并发处理由max_outstanding_messages控制。消息长度与编码参数max_message_len单条消息最大字节长度超长消息会被丢弃并报错小于 0 或未指定视为无限制。在 onMessage 中超长消息会先被Ack()避免反复重投并返回错误计入 accumulator 的错误统计。max_undelivered_messages如前文所述控制未投递消息上限默认1000。retry_delay_seconds接收器streaming pull发生意外错误后重启接收前的等待秒数默认5。base64_data为true时Telegraf 会在解析前尝试对消息体做 Base64 解码。许多向 PubSub 输出 JSON 的 GCP 服务会对负载做 Base64 编码如 Cloud Scheduler、审计日志推送等场景此开关即为此设计。content_encoding消息负载的压缩编码可设为gzip或identity不压缩。校验逻辑位于 Init()空值或identity视为不压缩gzip时通过internal.NewContentDecoder创建解码器其他取值直接报错invalid value ... for content_encoding。max_decompression_size当content_encoding非identity时限制解压后消息负载的最大字节数单位支持 B、KiB、KB、MiB、MB 等默认500MB。它既可调大以容纳更大的负载也可调小以防御解压炸弹decompression bomb攻击。消息处理链路从拉取到指标源码级插件启动后Start会拉起两个协程见 Start 实现一个执行receiveWithRetry持续从订阅拉消息另一个执行waitForDelivery监听投递确认。Stop()通过取消顶层 context 并等待两个协程退出完成优雅关闭。单条消息在onMessage中的处理顺序为长度校验超过max_message_len则 Ack 并报错解压decompressDataidentity直接透传gzip通过内容解码器解压且用互斥锁保护解码器并发安全Base64 解码decodeB64Data仅当base64_data true时执行base64.StdEncoding.DecodeString解析调用parser.Parse(data)生成指标解析失败或解析结果为空时立即 Ack避免无效消息反复投递其中空结果场景会通过sync.Once只输出一次NoMetricsCreated的调试日志注册跟踪将指标组加入 tracking accumulator消息存入undelivered表等待输出确认后再 Ack。receiveWithRetry则实现了故障自愈当Receive()返回错误且上下文未取消时记录错误日志、按retry_delay_seconds默认 5 秒等待后重启接收器循环直至成功或插件停止见 receiveWithRetry 实现。多订阅与多主题的消费方式插件假定你已经为主题创建好Pull 订阅。需要特别注意的是每个插件实例同一时刻只能监听一个订阅因此要同时从多个订阅/主题拉取消息必须配置多个插件实例即多个[[inputs.cloud_pubsub]]段。测试覆盖与可靠性验证仓库测试 cloud_pubsub_test.go 借助 subscription_stub_test.go 中的桩订阅stubSub与桩消息testMsg模拟 PubSub 行为验证了以下关键路径正常解析Influx 格式消息cpu_load_short,hostserver01 value23422.0 ...被正确解析为指标measurement、tag、field、时间戳全部校验Base64 解码base64_data true时对 Base64 编码后的消息可正确还原并解析gzip 解压content_encoding gzip时gzip 压缩负载可正确解压解析无效消息解析失败的消息会被立即 AckwaitForAck断言避免无谓重投超长消息超过max_message_len的消息被丢弃、记录错误并 Ack接收器异常Receive返回错误时按retry_delay_seconds记录错误并进入重试逻辑。这些测试同时验证了跟踪确认Ack 仅在指标交付后发生与错误处理两大可靠性设计可放心用于生产环境。快速开始建议在 GCP 上为主题创建 Pull 订阅并为本机或运行 Telegraf 的实例配置Application Default Credentials或准备服务账号凭据 JSON在配置文件中按上文完整示例填写project、subscription、data_format根据消息源的特征选择base64_dataJSON 负载常被 Base64 编码、content_encodinggzip 压缩等预处理选项结合代理的metric_batch_size合理设置max_undelivered_messages并可按需调整max_message_len、max_decompression_size等资源防护参数需要消费多个订阅时复制多个[[inputs.cloud_pubsub]]配置段即可。【免费下载链接】telegrafAgent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.项目地址: https://gitcode.com/GitHub_Trending/te/telegraf创作声明:本文部分内容由AI辅助生成(AIGC),仅供参考