Copy page

Bucket export

Have Costfluent deliver a workspace's FOCUS 1.4 cost rows every day into your own AWS S3, Google Cloud Storage, Azure Blob Storage or EU Object Storage bucket, with a credential that can only create files there.

A bucket export delivers a workspace's cost rows into a bucket you own, as FOCUS 1.4 files: one folder per billing day, in gzip-compressed CSV or in Parquet, with a manifest that says what the folder holds. Costfluent delivers each day once the workspace's data reaches it, and again whenever the day's rows change. To take one report's numbers into a spreadsheet once, export it instead; see Export to CSV or Excel.

Costfluent only creates files. It never reads, lists or deletes anything in the bucket. Every request goes to the storage host Costfluent builds from the destination's region, provider or storage account; no destination takes an address you type. Each destination has a credential of its own, separate from the connections that read your cloud costs:

Before you start

Included in the Pro, Business and Custom plans. Compare plans

• Adding, changing, testing or deleting a destination needs the Owner or Integration Owner organization role and Export costs on the workspace, so nobody sets up a feed of costs they could not export by hand. Anyone who can view the workspace sees its destinations and how far delivery reaches.
• A bucket for the feed, in AWS S3, Google Cloud Storage, Azure Blob Storage, or the Object Storage of OVHcloud, STACKIT, Scaleway, IONOS or Hetzner.
• A prefix: the folder in the bucket the feed goes under, such as finops. Costfluent writes everything under <prefix>/costfluent/, or under costfluent/ when you leave the prefix empty, and every grant below names that folder. A prefix is up to 200 characters of letters, digits and ! - _ . * ' ( ), in folders split by /, without ...
• The workspace's ID, which names its folder in the bucket: Settings, then Workspaces, lists it in the ID column.

Grant write access

Give the destination the one permission it needs, scoped to the export's folder. In the policies below, replace <bucket> with the bucket's name and <prefix> with your prefix; without a prefix, the resource is arn:aws:s3:::<bucket>/costfluent/*.

AWS S3

Costfluent writes as a role in your account. Its principal ARN and your organization's external ID are the same two an AWS connection's role trusts: Costfluent shows them when you add the destination and under Connect with the IAM console, and the Terraform provider's costfluent_aws_provider_info data source reads both. The role's trust policy:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "<Costfluent principal ARN>" },
      "Action": "sts:AssumeRole",
      "Condition": { "StringEquals": { "sts:ExternalId": "<your external ID>" } }
    }
  ]
}

Its permissions policy:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::<bucket>/<prefix>/costfluent/*"
    }
  ]
}

Costfluent also passes that permissions policy as the session policy every time it assumes the role, so its session can only put objects under the folder, whatever else the role allows.

Google Cloud Storage

Costfluent writes as the service account it created for your organization, the same one a Google Cloud connection shares its billing export with. Add the destination first (Add the destination): it creates the service account if your organization has none yet, and shows its address. Then grant that address Storage Object Creator (roles/storage.objectCreator) on the bucket, on the bucket's "Permissions" tab in the Google Cloud console or with:

Code
gcloud storage buckets add-iam-policy-binding gs://<bucket> \
  --member=serviceAccount:<service account address> \
  --role=roles/storage.objectCreator

The role creates objects but cannot read, list, replace or delete them. It covers the whole bucket, while Costfluent writes only under the export's folder; to keep the grant to the feed as well, give the feed a bucket of its own. If your organization restricts sharing to known domains, allow Costfluent's service account first, as a Google Cloud connection needs (Connect GCP).

Azure Blob Storage

Costfluent writes with a shared access signature (SAS) for the container. It accepts only a SAS whose permissions it can read from the token itself:

• a container SAS (sr=c), not an account SAS, and not one tied to a stored access policy;
• permissions Create, Write or both (sp=c, sp=w or sp=cw), and nothing else: a SAS that can also read, add, list or delete is refused;
• HTTPS only, and an expiry that has not passed.

In the Azure portal, open the container, select "Shared access tokens", tick only "Create" and "Write", set "Allowed protocols" to HTTPS only, choose the expiry, and generate the token. Copy the "Blob SAS token", the part after the ?, not the SAS URL. With the Azure CLI:

Code
az storage container generate-sas \
  --account-name <storage account> \
  --name <container> \
  --permissions cw \
  --https-only \
  --expiry <YYYY-MM-DDTHH:MMZ> \
  --output tsv

S3-compatible Object Storage

Choose the provider and enter the bucket's region. Together they form the only host Costfluent writes to:

Create an access key for the feed and limit it to putting objects under the export's folder, with this policy on the key's user, or as a bucket policy statement that names that user as its principal, whichever the provider offers:

Code
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::<bucket>/<prefix>/costfluent/*"
    }
  ]
}

Add the destination

The key pair and the SAS are stored encrypted and are never shown again; to change one, enter a new one. An AWS or Cloud Storage destination stores no secret.

Test objects

Each test writes <prefix>/costfluent/_costfluent-check/<yyyyMMddTHHmmssZ>.txt, named by the instant of the test in UTC, so tests never overwrite each other or a delivery. The object holds one line saying that Costfluent wrote it to check the folder and that it is safe to delete. Costfluent never deletes it; delete it whenever you like.

For AWS, the test also asks AWS to let Costfluent assume the role with an external ID that is not your organization's. The role must refuse: a role that lets Costfluent in without your external ID would let any Costfluent customer write to your bucket, so the test fails until the trust policy requires it.

The test runs in the background, and its result appears on the destination after a moment. A failed test names its reason: the write was refused, the bucket was not found, the credential was refused, the role does not require your external ID, the SAS has expired, or the storage did not answer.

What lands in the bucket

Key layout

Code
<prefix>/costfluent/<workspace ID>/focus-1.4/billing-date=<YYYY-MM-DD>/version=<yyyyMMddTHHmmssZ>/part-00000.csv.gz
<prefix>/costfluent/<workspace ID>/focus-1.4/billing-date=<YYYY-MM-DD>/version=<yyyyMMddTHHmmssZ>/part-00001.csv.gz
<prefix>/costfluent/<workspace ID>/focus-1.4/billing-date=<YYYY-MM-DD>/version=<yyyyMMddTHHmmssZ>/manifest.json

• The workspace's folder is its ID (wsp_…), never its name, so renaming the workspace does not move the feed.
• billing-date is the day the rows' charge period starts (ChargePeriodStart), in UTC.
• A Parquet destination writes part-00000.parquet and so on instead of .csv.gz.
• A part holds at most 1,000,000 rows, numbered from part-00000. The manifest is written last.

Versions

Each delivery of a day lands in a new version= folder, named by the UTC instant Costfluent reserved for it before its first upload. To read the feed, take for each billing day the newest version= folder that holds a manifest.json, and ignore the others:

• A folder without a manifest is not complete yet. If a run stops partway, the next run finishes the same folder, writing the same keys, rather than starting another.
• A day is delivered again, under a newer version, only when its rows changed: a provider restated the day, or the workspace's integrations or accounts changed. A day whose rows did not change is not written again.
• A day restated to nothing gets a newer version whose manifest lists no parts and "rows": 0.

Costfluent never deletes an older version. Keep them for history or remove them yourself.

The manifest

manifest.json says what the version folder holds:

Code
{
  "focusVersion": "1.4",
  "workspaceId": "wsp_…",
  "billingDate": "2026-10-25",
  "version": "20261026T080000Z",
  "format": "csv",
  "rows": 1834,
  "columns": ["BillingPeriodStart", "BillingPeriodEnd", "BillingCurrency", "…"],
  "parts": [
    {
      "key": "finops/costfluent/wsp_…/focus-1.4/billing-date=2026-10-25/version=20261026T080000Z/part-00000.csv.gz",
      "rows": 1834,
      "bytes": 51840,
      "sha256": "…"
    }
  ]
}

• format is csv or parquet, and columns lists every column of the parts in their order.
• rows is the day's row count, the sum of its parts' rows.
• Each part's sha256 is the lower-case hex SHA-256 of the bytes uploaded, the compressed file for CSV, so a consumer can check that it read the whole part.

The files

The rows are the workspace's cost rows for the day, as Costfluent stores them: from the integrations the workspace holds, within the accounts it covers. The columns are the FOCUS 1.4 columns, named as FOCUS names them (BilledCost, ChargePeriodStart), followed by Costfluent's own columns, which carry the x_ prefix FOCUS gives custom columns (x_AmortizedCost). Amounts are in the billing and pricing currencies FOCUS defines; the workspace's currency settings, segments and virtual tags do not apply.

• CSV: gzip-compressed UTF-8, as RFC 4180 defines it: a header row of the column names, lines ending in CRLF, a field quoted only when it holds a comma, a quote or a line break, with its quotes doubled. Amounts are written exactly as stored, with up to 12 decimal places, timestamps as yyyy-MM-ddTHH:mm:ssZ in UTC, and Tags and AllocatedTags as JSON objects. An empty field is a missing value.
• Parquet: the same columns, typed: amounts as DECIMAL(38, 12), timestamps as UTC timestamps, everything else as strings, the tag columns as their JSON objects. A row group holds at most 100,000 rows.

How it behaves

• Delivery runs every four hours. Each run delivers at most 31 days to a destination, oldest first, so a new destination fills its history over several runs.
• The days run from the first day the plan's history keeps to the workspace's data-through day, the latest day every connection of the workspace has delivered, the day reports show as "Data through". A day after it waits until every connection has delivered it, and a day whose rows are still being processed waits for the next run, so a day is never delivered half.
• A day without cost rows gets no folder: a missing day means Costfluent holds no cost for it, not a cost of zero.
• The destination shows how far delivery reaches: not delivered yet; delivered, with the day it is delivered through; behind, when that day is more than two days before today, as during a backfill or while a connection's data has stopped arriving; or failing, with the reason of the last refused write. A failing destination is tried again on the next run.
• Changing the prefix or the format starts delivery over from the first day under the new folder or in the new format. The files already delivered stay where they are. A different bucket is a different destination: delete this one and add another.
• Deleting a destination stops its delivery and deletes its stored key pair or SAS. The files stay in the bucket. Remove the role, the grant or the key on your side as well; Costfluent's service account for Cloud Storage remains while your organization exists, so its grant outlives a deleted destination until you remove it.

Related

• Export to CSV or Excel
• Connect AWS
• Connect GCP
• Power BI and Excel
• Plans and billing