Copy page

Connect a source

Connect GCP

Share your BigQuery billing export with the service account Costfluent created for your organization, in the console or with Terraform, and add the billing account as a connection.

Costfluent reads Google Cloud cost from your BigQuery billing export. It creates a service account for your organization in its own Google Cloud project, and you grant that account read access to the export dataset. No key is created or exchanged, and nothing is created in your cloud except that one grant.

Connect a billing account, not a project. Its export already carries the cost of every project the billing account pays for, so one connection covers them all.

How Costfluent connects

BigQuery Data Viewer on the export dataset, and nothing else. Costfluent runs its queries in its own project and pays for them there, so it needs no role in your project and cannot see any other dataset, run jobs on your bill, or change anything.

For the carbon footprint report, also grant the service account BigQuery Data Viewer on the dataset your Carbon Footprint export writes to (Carbon Footprint export).

To list your committed use discounts under Commitments, also grant the service account BigQuery Data Viewer on the dataset your committed use discount metadata export writes to (Committed use discount export).

Recommender roles

To have Savings list Google Cloud Recommender's cost recommendations, also grant the service account these roles on each project to read, or once on a folder or the organization above them, which every project under it inherits. They read recommendations and insights only:

The broader Recommender Viewer role is not needed; it would also show the project's IAM, firewall and security insights. A project without these roles is skipped (Savings).

Prerequisites

1. The Owner or Integration owner organization role in Costfluent.
2. A project to hold the billing data. A dedicated one keeps access narrow.
3. A BigQuery dataset in that project, in the US or EU multi-region.
4. Under Billing, then Billing export, the Detailed usage cost export enabled into that dataset. The standard export lacks the per-resource fields Costfluent reads. Google can take a few hours to write the first rows.
5. Optional, for the carbon footprint report: the Carbon Footprint export to BigQuery set up for the same billing account, into a dataset in the same project.
6. Optional, for committed use discounts on Commitments: under Billing, then Billing export, the committed use discount metadata export enabled for the same billing account, into a new dataset in the same project. Google's setup creates that dataset and says not to use an existing one. The export is a Preview feature of Cloud Billing, and enabling it needs the Billing Account Administrator role on the billing account and BigQuery Admin on the project.

Connect

Connect in the console

Costfluent validates the query it will run before storing anything, so a connection that saves can read your cost. If Google does not answer during that check, the connection is saved anyway and checked again automatically.

Connect with Terraform

With the Costfluent Terraform provider, the whole connection is one apply. The costfluent_gcp_service_account resource returns the service account's email, the costfluent/cost-access/gcp module grants it the role on the dataset, and the module's credentials output feeds costfluent_provider:

Code
resource "costfluent_gcp_service_account" "this" {}

module "costfluent_gcp" {
  source  = "costfluent/cost-access/gcp"
  version = "~> 0.4"

  project_id                       = "<project holding the export>"
  billing_dataset_id               = "<export dataset>"
  billing_account_id               = "012345-ABCDEF-678901"
  costfluent_service_account_email = costfluent_gcp_service_account.this.email

  # Optional: the projects whose cost recommendations Savings lists, or the numeric IDs of folders
  # or the organization whose projects it lists.
  recommender_project_ids     = ["<project>", "<another project>"]
  recommender_folder_ids      = []
  recommender_organization_id = null

  # Optional: the Carbon Footprint export's dataset, in the same project.
  carbon_dataset_id = "<carbon footprint dataset>"

  # Optional: the committed use discount metadata export's dataset, in the same project.
  cud_dataset_id = "<committed use discount dataset>"
}

resource "costfluent_provider" "gcp" {
  key          = "gcp"
  display_name = "Google Cloud"
  credentials  = module.costfluent_gcp.credentials
  settings     = module.costfluent_gcp.settings

  depends_on = [module.costfluent_gcp]
}

credentials carries the billing account ID, the project and the dataset, the same three values the dialog asks for. settings carries carbon_dataset when carbon_dataset_id is set and cud_dataset when cud_dataset_id is set, and is empty without either. The module grants the service account BigQuery Data Viewer on each dataset you name. A new grant takes a little while to apply, so costfluent_provider retries for up to three minutes while Costfluent reports the dataset as not shared yet. The resource also returns organization_id and directory_customer_id, the IDs to allow when your organization restricts sharing to known domains.

recommender_folder_ids and recommender_organization_id grant the three Recommender roles on folders or the organization instead of each project, so projects created under them later are covered too. Costfluent reads recommendations only for the projects in the billing export's recent cost rows, whichever way the roles are granted.

Data and freshness

Connecting imports as far back as your plan's backfill reaches, and at most six months (Manage connections), as long as the export holds data that far back.

Google corrects a past month by adding rows to the export, with no time limit. Every sync checks the export for rows added to closed months, and Costfluent collects each corrected month again, as far back as your plan's history reaches and at most three years.

Carbon Footprint export

Google delivers each month's emissions on the 15th of the next month, and the export's first delivery reaches back to January 2021. Costfluent reads it on the 22nd (Carbon footprint), with the same service account as the cost. The Carbon dataset is set when you add the connection; the connection page cannot change it. To add one to an existing connection, set carbon_dataset through Terraform's settings or the Public API's PUT /v1/providers/{providerId}.

Committed use discount export

Google writes a daily snapshot of the billing account's spend-based committed use discounts to the export's cud_subscriptions_export table. Costfluent reads it once a day, with the same service account as the cost, and lists each commitment under Commitments held with when it ends. As one nears its end, every workspace that lists it gets a notification (Commitments). The Committed use discount dataset is set when you add the connection; the connection page cannot change it. To add one to an existing connection, set cud_dataset through Terraform's settings or the Public API's PUT /v1/providers/{providerId}.

Reporting dimensions

What each group-by dimension of a cost report holds for a Google Cloud connection, and the column of the detailed usage cost export it comes from.

Update or disconnect

After connecting

The Manage tab lists the connection by its display name, with its Billing account ID, Project and Dataset. Open the gear icon beside it to see whether the dataset is still readable, which workspaces the connection feeds, and which billing months have been imported.

Run checks on the connection page runs the checks again on request and reports each one separately, so a failure names the step that failed.

Rotating or removing access

There is no key to rotate. To retire the connection, remove it in Costfluent and remove the service account's grant on the dataset, or run terraform destroy. Removing it erases the cost it imported (Remove a connection). When a Costfluent organization is deleted, its service account is deleted with it.

Costfluent also deletes the service account once no connection and no Cloud Storage destination has used it for 7 days. Selecting "Add billing account", provisioning it through the API, or a Terraform plan creates it again at the same address, but Google does not carry the old account's grants over to it: grant it everything again, as in the steps above, before you connect.

Troubleshooting

• The dataset is not shared yet. The test names the dataset and the service account. Grant the role, and try again a few minutes later, since a new grant can take that long to apply.
• The service account was removed. If you connect after Costfluent deleted the unused service account (Rotating or removing access), Costfluent creates it again and the test says so. Grant it BigQuery Data Viewer on the dataset again, and on the carbon and committed use discount datasets and the Recommender roles if you used them (applying the Costfluent module again gives all of them), then connect.
• The billing export is not populated yet. The dataset exists but has no export table. Google creates it a few hours after the Detailed usage cost export is enabled.