Cloud Dataflow is a fully-managed service for transforming and enriching data in stream (real-time) and batch (historical) modes with equal reliability and expressiveness – no more complex workarounds or compromises needed. With its serverless approach to resource provisioning and management, you have access to virtually limitless capacity to solve your biggest data processing challenges, while paying only for what you use.
Cloud Dataflow unlocks transformational use cases across industries, including:
- Check Clickstream, Point-of-Sale, and segmentation analysis in retail.
- Check Fraud detection in financial services.
- Check Personalized user experience in gaming.
- Check IoT analytics in manufacturing, healthcare, and logistics.
Setup
To set up the Google integration and discover the Google service,
go to Google Integration Discovery Profile and select GOOGLE/Dataflow Job.
Supported metrics
| New OpsRamp Metric | Google Metric | Metric Display Name | Unit | Description |
|---|---|---|---|---|
| google_dataflow_job_backlog_bytes | dataflow.googleapis.com/job/backlog_bytes | Per-stage backlog in bytes | bytes | Amount of known, unprocessed input for a stage, in bytes. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_backlog_elements | dataflow.googleapis.com/job/backlog_elements | Per-stage backlog in elements | count | Amount of known, unprocessed input for a stage, in elements. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_billable_shuffle_data_processed | dataflow.googleapis.com/job/billable_shuffle_data_processed | Billable shuffle data processed | bytes | The billable bytes of shuffle data processed by this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_bundle_user_processing_latencies | dataflow.googleapis.com/job/bundle_user_processing_latencies | Bundle user processing latencies | ms | Bundle user processing latencies from a particular stage. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_current_num_vcpus | dataflow.googleapis.com/job/current_num_vcpus | Current number of vCPUs in use | count | The number of vCPUs currently being used by this Dataflow job. This is the current number of workers times the number of vCPUs per worker. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_current_shuffle_slots | dataflow.googleapis.com/job/current_shuffle_slots | Current shuffle slots in use | count | The current shuffle slots used by this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_data_watermark_age | dataflow.googleapis.com/job/data_watermark_age | Data watermark lag | seconds | The age (time since event timestamp) up to which all data has been processed by the pipeline. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_disk_space_capacity | dataflow.googleapis.com/job/disk_space_capacity | Disk Space Capacity | bytes | The amount of persistent disk currently being allocated to all workers associated with this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_dofn_latency_average | dataflow.googleapis.com/job/dofn_latency_average | Average message processing time per DoFn. | ms | The average processing time for a single message in a given DoFn (over the past 3 min window). Note that this includes time spent in GetData calls. Available for jobs running on Streaming Engine on the Legacy Runner. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_dofn_latency_max | dataflow.googleapis.com/job/dofn_latency_max | Maximum message processing time per DoFn. | ms | The maximum processing time for a single message in a given DoFn (over the past 3 min window). Note that this includes time spent in GetData calls. Available for jobs running on Streaming Engine on the Legacy Runner. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_dofn_latency_min | dataflow.googleapis.com/job/dofn_latency_min | Minimum message processing time per DoFn. | ms | The minimum processing time for a single message in a given DoFn (over the past 3 min window). Available for jobs running on Streaming Engine on the Legacy Runner. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_dofn_latency_num_messages | dataflow.googleapis.com/job/dofn_latency_num_messages | Number of messages processed per DoFn. | count | The number of messages processed by a given DoFn (over the past 3 min window). Available for jobs running on Streaming Engine on the Legacy Runner. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_dofn_latency_total | dataflow.googleapis.com/job/dofn_latency_total | Total message processing time per DoFn. | ms | The total processing time for all messages in a given DoFn (over the past 3 min window). Note that this includes time spent in GetData calls. Available for jobs running on Streaming Engine on the Legacy Runner. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_duplicates_filtered_out_count | dataflow.googleapis.com/job/duplicates_filtered_out_count | Duplicate message count per stage | count | The number of messages being processed by a particular stage that have been filtered out as duplicates. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_elapsed_time | dataflow.googleapis.com/job/elapsed_time | Elapsed time | seconds | Duration that the current run of this pipeline has been in the Running state so far, in seconds. When a run completes, this stays at the duration of that run until the next run starts. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_element_count | dataflow.googleapis.com/job/element_count | Element count | count | Number of elements added to the pcollection so far. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_estimated_byte_count | dataflow.googleapis.com/job/estimated_byte_count | Estimated byte count | bytes | An estimated number of bytes added to the pcollection so far. Dataflow calculates the average encoded size of elements in a pcollection and mutiplies it by the number of elements. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_estimated_bytes_active | dataflow.googleapis.com/job/estimated_bytes_active | Active Size | bytes | Estimated number of bytes active in this stage of the job. |
| google_dataflow_job_estimated_bytes_consumed_count | dataflow.googleapis.com/job/estimated_bytes_consumed_count | Throughput | bytes | Estimated number of bytes consumed by the stage of this job. |
| google_dataflow_job_estimated_bytes_produced_count | dataflow.googleapis.com/job/estimated_bytes_produced_count | Estimated Bytes Produced | count | The estimated total byte size of elements produced by each PTransform. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_is_failed | dataflow.googleapis.com/job/is_failed | Failed | count | A value of 1 indicates that the job has failed. This metric isn't recorded for jobs that fail before launch. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_max_worker_instances_limit | dataflow.googleapis.com/job/max_worker_instances_limit | Autoscaling worker instances ceiling | count | The maximum number of workers autoscaling is allowed to request. |
| google_dataflow_job_memory_capacity | dataflow.googleapis.com/job/memory_capacity | Memory Capacity | bytes | The amount of memory currently being allocated to all workers associated with this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_min_worker_instances_limit | dataflow.googleapis.com/job/min_worker_instances_limit | Autoscaling worker instances flooring | count | The minimum number of workers autoscaling is allowed to request. |
| google_dataflow_job_oldest_active_message_age | dataflow.googleapis.com/job/oldest_active_message_age | Oldest active message processing time per DoFn. | ms | How long the oldest active message in a DoFn has been processing for. Available for jobs running on Streaming Engine on the Legacy Runner. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_per_stage_data_watermark_age | dataflow.googleapis.com/job/per_stage_data_watermark_age | Per-stage data watermark lag | seconds | The age (time since event timestamp) up to which all data has been processed by this stage of the pipeline. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_per_stage_system_lag | dataflow.googleapis.com/job/per_stage_system_lag | Per-stage system lag | seconds | The current maximum duration that an item of data has been processing or awaiting processing in seconds, per pipeline stage. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_processing_parallelism_keys | dataflow.googleapis.com/job/processing_parallelism_keys | The approximate number of parallel processing keys | count | Approximate number of keys in use for data processing for each stage. Processing for any given key is serialized, so the total number of keys for a stage represents the maximum available parallelism at that stage. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_pubsub_late_messages_count | dataflow.googleapis.com/job/pubsub/late_messages_count | Job Pubsub Late Messages Count | count | The number of messages from Pub/Sub with timestamp older than the estimated watermark. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_pubsub_published_messages_count | dataflow.googleapis.com/job/pubsub/published_messages_count | Job Pubsub Published Messages Count | count | The number of Pub/Sub messages published broken down by topic and status. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_pubsub_pulled_message_ages | dataflow.googleapis.com/job/pubsub/pulled_message_ages | Job Pubsub Pulled Message Ages | ms | The distribution of pulled but unacked Pub/Sub message ages. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_pubsub_read_count | dataflow.googleapis.com/job/pubsub/read_count | PubsubIO.Read requests from Dataflow jobs | count | Pub/Sub Pull Requests. For Streaming Engine, this metric is deprecated. See the "Using the Dataflow monitoring interface" page for upcoming changes. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_pubsub_streaming_pull_connection_status | dataflow.googleapis.com/job/pubsub/streaming_pull_connection_status | Job Pubsub Streaming Pull Connection Status | % | Percentage of all Streaming Pull connections that are either active (OK status) or terminated because of an error (non-OK status). When a connection is terminated, Dataflow will wait some time before attempting to re-connect. For Streaming Engine only. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_pubsub_write_count | dataflow.googleapis.com/job/pubsub/write_count | Job Pubsub Write Count | count | Pub/Sub Publish requests from PubsubIO.Write in Dataflow jobs. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_pubsub_write_latencies | dataflow.googleapis.com/job/pubsub/write_latencies | Job Pubsub Write Latencies | ms | Pub/Sub Publish request latencies from PubsubIO.Write in Dataflow jobs. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_streaming_engine_key_processing_availability | dataflow.googleapis.com/job/streaming_engine/key_processing_availability | Current processing key-range availability | % | Percentage of streaming processing keys that are assigned to workers and available to perform work. Work for unavailable keys will be deferred until keys are available. |
| google_dataflow_job_streaming_engine_persistent_state_read_bytes_count | dataflow.googleapis.com/job/streaming_engine/persistent_state/read_bytes_count | Storage bytes read | count | Storage bytes read by a particular stage. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_streaming_engine_persistent_state_stored_bytes | dataflow.googleapis.com/job/streaming_engine/persistent_state/stored_bytes | Current persistence state usage | bytes | Current bytes stored in persistent state for the job. |
| google_dataflow_job_streaming_engine_persistent_state_write_bytes_count | dataflow.googleapis.com/job/streaming_engine/persistent_state/write_bytes_count | Storage bytes written | count | Storage bytes written by a particular stage. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_streaming_engine_persistent_state_write_latencies | dataflow.googleapis.com/job/streaming_engine/persistent_state/write_latencies | Storage write latencies | ms | Storage write latencies from a particular stage. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_streaming_engine_stage_end_to_end_latencies | dataflow.googleapis.com/job/streaming_engine/stage_end_to_end_latencies | Per stage end to end latencies. | ms | Distribution of time spent by streaming engine in each stage of the pipeline. This time includes shuffling messages, queueing them for processing, processing, queueing for persistent state write, and the write itself. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_system_lag | dataflow.googleapis.com/job/system_lag | System lag | seconds | The current maximum duration that an item of data has been processing or awaiting processing, in seconds. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_target_worker_instances | dataflow.googleapis.com/job/target_worker_instances | Target Worker Instances | count | The desired number of worker instances. |
| google_dataflow_job_timers_pending_count | dataflow.googleapis.com/job/timers_pending_count | Timers pending count per stage | count | The number of timers pending in a particular stage. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_timers_processed_count | dataflow.googleapis.com/job/timers_processed_count | Timers processed count per stage | count | The number of timers completed by a particular stage. Available for jobs running on Streaming Engine. Sampled every 60 seconds. After sampling, data is not visible for up to 60 seconds. |
| google_dataflow_job_total_dcu_usage | dataflow.googleapis.com/job/total_dcu_usage | Total DCU usage | compute units | The total amount of DCUs (Data Compute Unit) used by the Dataflow job since it was launched. |
| google_dataflow_job_total_memory_usage_time | dataflow.googleapis.com/job/total_memory_usage_time | Total memory usage time | GB-seconds | The total GB seconds of memory allocated to this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_total_pd_usage_time | dataflow.googleapis.com/job/total_pd_usage_time | Total PD usage time | GB-seconds | The total GB seconds for all persistent disk used by all workers associated with this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_total_shuffle_data_processed | dataflow.googleapis.com/job/total_shuffle_data_processed | Total shuffle data processed | bytes | The total bytes of shuffle data processed by this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_total_streaming_data_processed | dataflow.googleapis.com/job/total_streaming_data_processed | Total streaming data processed | bytes | The total bytes of streaming data processed by this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_total_vcpu_time | dataflow.googleapis.com/job/total_vcpu_time | Total vCPU time | seconds | The total vCPU seconds used by this Dataflow job. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
| google_dataflow_job_user_counter | dataflow.googleapis.com/job/user_counter | User Counter | count | A user-defined counter metric. Sampled every 60 seconds. After sampling, data is not visible for up to 240 seconds. |
Event support
- Supported
- Configurable in OpsRamp Google Integration Discovery Profile.