Introduction

As streaming analytics and machine learning needs continue to grow, customers with predictable processing volumes want to better optimize their Dataflow costs. Today, Google introduced the general availability of Dataflow streaming committed use discounts (CUDs), providing a new way for you to save money on a key driver of your streaming costs: streaming compute. By committing to a baseline amount of Dataflow streaming it offers two levels of discounts, depending on the commitment period:

• A one-year CUD gives you a 20% discount from the on-demand rate.
• A three-year CUD gives you a 40% discount from the on-demand rate.

Dataflow streaming CUDs are spend-based commitments, and apply to the following Dataflow resources across all projects or regions that are associated with a single Cloud Billing account:

• Worker CPU and memory for streaming jobs,
• Streaming Engine data processed,
• Streaming Engine compute units,
• Data Compute Units (DCUs) for streaming jobs.

Dataflow CUDs don’t apply to your spending on the following resources:

• Worker CPU and memory for batch and FlexRS jobs,
• Dataflow Shuffle data processed,
• Data Compute Units (DCUs) for batch jobs,
• Persistent Disk storage,
• GPUs,
• Snapshots,
• Confidential VMs.

Recommendations for choosing a commitment

When you consider the purchase of Dataflow CUDs for streaming jobs, consider these factors:

• Region: Dataflow CUDs apply to all streaming jobs in a project across regions. They apply to both regional and multi-region configurations. If you have Dataflow streaming jobs in multiple regions, calculate the expenditure across all of the regions that your projects use when making a commitment.
• Projects: Determine the consistent baseline expenditure per project when calculating total commitment. Consider that production streaming jobs usually run 100% of the time, whereas development or staging jobs might run intermittently.
• Jobs: If your Dataflow streaming jobs use a steady number of resources around the clock, consider purchasing a commitment that covers all of your resource usage. However, if you frequently scale your resources up or down, either manually or by using autoscaling, consider purchasing commitments only for your baseline predictable usage.

If you have jobs that you run only for bursts or brief durations, exclude them from your calculations.

Your commitment fee applies to every hour during the term of the commitment, regardless of actual usage. Choose your commitment amount based on both your historical resource usage for Dataflow streaming jobs and your future expectations.

How Rabbit helps

Managing SKUs eligible for Dataflow CUDs can be complex, with some qualifying and others not. Additionally, the dynamic nature of autoscaling requires careful cost considerations for every minute. Rabbit steps in to streamline this process with its CUD Planner.

Screenshot is from followrabbit.ai’s CUD planner feature

By analyzing your past usage data, Rabbit offers three practical strategies for planning your commitment:

1. Cover minimum stable usage: This strategy identifies the lowest hourly usage within the selected period and suggests a commitment based on that benchmark.
2. Maximize savings: Dataflow’s scalability often results in peak and off-peak usage periods. Committing slightly above the minimum usage can lead to significant savings. As the system scales up, the savings outweigh any costs from overcommitment during periods of scale-down.
3. Custom commitment: Rabbit provides the flexibility to tailor commitments, allowing users to choose a commitment percentage ranging from zero usage to maximizing savings.

Screenshot is from followrabbit.ai’s CUD planner feature

Take the next step

Try this feature free for the first 30 days with a quick 10-minute onboarding via the Google Marketplace. For more information about Dataflow CUD go to the documentation.