This document will guide you through:
Setting up a custom billing datasource on Costory
What you will need to configure Costory
Big Query Table Path
Context
Costory comes with a number of ready-to-use connectors (AWS, Azure, Datadog, and more). However, you may also want to track costs from systems for which we don’t yet provide a connector. In those cases, Costory supports the ingestion of custom billing data sources in the FOCUS format via a BQ Table
This guide will walk you through how to set it up.
⚠️ Please note: setting up a custom datasource requires some data pipeline configuration on your side.
Preparation Work
You will need to create a data pipeline that feed usage/billing data from your source at regular interval depending on the data availability on the source (daily, weekly, monthly) and output it in BQ in a EU dataset of your choice
DB Schema
Here is the expected schema and types
Note : In the setup wizard, you will be able to rename fields (but not change the type)
{
"fields": [
{
"mode": "NULLABLE",
"name": "availability_zone",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "billing_account_id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "billing_currency",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "charge_category",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "charge_description",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "charge_frequency",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "commitment_discount_category",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "commitment_discount_id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "commitment_discount_type",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "consumed_quantity",
"type": "FLOAT"
},
{
"mode": "NULLABLE",
"name": "consumed_unit",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "effective_cost",
"type": "FLOAT"
},
{
"mode": "NULLABLE",
"name": "invoice_issuer",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "list_cost",
"type": "FLOAT"
},
{
"mode": "NULLABLE",
"name": "list_unit_price",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "pricing_category",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "pricing_quantity",
"type": "FLOAT"
},
{
"mode": "NULLABLE",
"name": "pricing_unit",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "provider",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "publisher",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "region_id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "resource_id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "resource_name",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "resource_type",
"type": "INTEGER"
},
{
"mode": "NULLABLE",
"name": "category_focus",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "service_name",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "sku_id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "sku",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "sku_price_id",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "sub_account_id",
"type": "STRING"
},
{
"fields": [
{
"mode": "NULLABLE",
"name": "key",
"type": "STRING"
},
{
"mode": "NULLABLE",
"name": "value",
"type": "STRING"
}
],
"mode": "REPEATED",
"name": "labels",
"type": "RECORD"
},
{
"mode": "NULLABLE",
"name": "charge_period_start",
"type": "TIMESTAMP"
},
{
"mode": "NULLABLE",
"name": "charge_period_end",
"type": "TIMESTAMP"
},
{
"mode": "NULLABLE",
"name": "billing_period_start",
"type": "DATETIME"
},
{
"mode": "NULLABLE",
"name": "billing_period_end",
"type": "DATETIME"
},
{
"mode": "NULLABLE",
"name": "billed_cost",
"type": "FLOAT"
},
{
"mode": "NULLABLE",
"name": "datasource_costory",
"type": "STRING"
}
]
}
Permissions
Add the following permissions to your costory provided service account. More details in Step 2 of the GCP Cloud Platform Setup Guide
BigQuery Data viewer: to be able to read all the tables in the dataset.BigQuery metadata viewer:to be able to list tables in the dataset.
Setup the datasource on Costory
The table name should look like: <project_id>:<dataset_id>.<table_name>.
Click on Import New Billing Datasource
Select Custom Billing Format (FOCUS)
In Select datasource
Enter the table path
Provider: the name of the datasource you want to import. This is the label you’ll see in the Provider dropdown (e.g.,
OVH)Billing_account_id (optional): use this if you need to override the default account identifier. This is useful when importing data from multiple accounts under the same provider
In Columns Mappings you will be able to rename fields to FOCUS columns