Zoltán Guth, CTO of Rabbit shares the team’s findings and experiences with our customers on how to identify cost-saving opportunities at the service level.

With 10+ years of experience with Google Cloud and helping dozens of customers to optimize performance and costs in multiple areas of GCP, we recognized that cloud engineers often lack the tools needed for transparency and optimization of cloud systems.

Working with some of the largest cloud users and understanding their needs brought Rabbit to life, which not only offers detailed visibility into cloud usage but also provides automation tools to help teams optimize their cloud resources efficiently.

In the first part of the ultimate guide for GCP cost optimization, we covered Google Kubernetes Engine, Google Compute Engine and Committed Use Discounts, now let’s dive deeper into BigQuery and Cloud Storage!

BigQuery

1. Compute Cost
   1. Choosing between on-demand and slot-based models can be complex. For users spending over $30k monthly, slot-based models often make sense, especially with PB-scale data. However, a mixed model can sometimes offer savings of $10k-$50k depending on usage patterns.
   2. Optimizing max-slot settings is crucial. In some parts of the day, you may need more slots for your reservations to deal with your critical jobs, but for the other part, this can lead to waste because BQ can scale up too eagerly leading to not well-utilized slots. One customer reduced waste by $30k monthly by adopting a more dynamic max-slot setting throughout the day. This is the part where Rabbit can help you automatically adjust max slot settings based on the historical usage to fulfil mixed needs.
   3. Base slots with 1-year (20% discount) or 3-year (40% discount) commitment may make sense even if you are not using it for the whole period because of the discount. Rescheduling queries to flatten usage throughout the day can also help.
   4. Proper clustering and partitioning can reduce the data read and slot usage, further lowering costs.
   5. For query optimisation, this is a good starting point on what to do.
2. Storage cost
   1. You can be billed for BigQuery data storage in either logical or physical (compressed) bytes, or a combination of both. Switching from a logical to a physical pricing model can lead to significant savings. For example, a customer reduced costs from $260k to $100k monthly.
   2. It’s kind of typical that in development environments or after a migration there are leftover tables that can add up to a significant cost for a bigger company. Detecting and removing leftover tables can help to reduce storage costs.
   3. If you have a table that is not modified for 90 consecutive days, the price of storage for that table automatically drops by 50%. If you have a partitioned table, each partition is considered separately for eligibility for long-term pricing, subject to the same rules as non-partitioned tables. Therefore it can save up to 50% if you have partitioning on tables that you are constantly inserting data into.

Cloud Storage

1. Implement lifecycle rules to automatically remove old objects or move them to a different class. For example, keeping only the three most recent versions of objects can also reduce costs.
2. Consider using autoclass if your data access patterns are heterogeneous. Be aware of this feature if you are already using multiple class types for your bucket. When you turn this feature on every file is automatically moved back to the Standard class type, and for instance, for Archive class type you have to wait for a year to have your files again in that class type if it wasn’t accessed during that period.
3. For large buckets, turning on storage insights inventory reports can help identify the largest folders/files, aiding in cost management.
4. To optimize your class types further you can turn on the usage logs to see how frequently you are accessing different files. Based on this and the inventory reports you can set up more advanced lifecycle rules like moving files to archive class type if it wasn’t read for 30 days and are older than 60 days. Rabbit helped save $15k on a monthly basis for a customer by automatically changing the class type of files based on those rules.

Finding all the optimization options is difficult, but Rabbit is here to help.

or contact us to try Rabbit on demo data.