This document will help you
comprehensive instructions for setting up Costory to connect with your Datadog account for cost management
how to generate and provide the necessary access keys.
You will only need to provide in Costory:
API Key
Application Key
Region
Known LImitation
See here for Datadog Enteprise customers purchased via AWS marketplace
Setup
1. Log into Your Datadog Account
Access your Datadog account as usual.
2. Create an API Key
Navigate to Organization Settings by hovering over your user profile (bottom left corner).
Select API Keys.
Click on Create API Key and name it (e.g., costory).
Important: Save the API key value securely in your password manager.
3. Create an Application Key
In the Organization Settings, go to Application Keys.
Click on Create Application Key and name it (e.g., costory).
After creating the key, click Edit on the application key to set the necessary scope:
usage_read: This grants access to view usage and billing data, but not contract details, commit details, or logs.billing_read: To access cost estimation.ci_visibility_read: if you want metric visibility (optional)timeseries_query: Optional, If you want to use metric timeseries (optional)
Important: Save the application key value securely in your password manager.
4. Run the Datadog Integration on Costory
Go to Settings > Integration on Costory
Select Datadog and click on Add Datadog Configuration
Run thru the Datatadog wizard with the information provided
Name your Datadog Configuration and select the correct region from the dropdown (e.g. datadoghq.eu for EU customers)
Then enter your API key and Application Key and click on "Save Config"
If the API and Applications keys are valid - you will see a notification confirming the successful Datadog integration.
Notes
If the validation fail, check the region field. This is the most common mistake
The data will take a big of time to be fetched.
Data Insights and Considerations
Billing Frequency and Daily Cost Calculation:
Datadog bills on a monthly basis, meaning costs are generally not available on a daily basis. For example:
Metrics: Billed based on the monthly average of unique custom metrics submitted per hour, making it difficult to determine the cost impact of a single hour.
Logs Ingested: Billed based on the total GB ingested over the month, allowing for an accurate daily cost by dividing the total.
Cost Types in Datadog:
There are two types of cost in DataDog:
Committed Costs: Billed on the first day of the month and applied until usage exceeds the commitment.
On-Demand Costs: Calculated daily, with the cost for each day being the difference between the current estimate and the previous day's estimate (note that this data is typically 72 hours behind).
Additional tags:
How does Costory transform the data
On-Demand Costs
For the on demand cost, Costory receives using estimated cost api the estimated cost every day, Costory considers the daily cost is the difference between the cost of the day before and today (data is always 72 hours late). This means that you can see negative costs on a daily basis: often this happens if you increased the number of metrics sent to datadog on one day and then removed them (It is caused by the way Datadog bills some services: average over a month…)
Committed Costs:
However for committed costs it is billed the first day of the month and active until the committed usage is lower than actual usage.
There are two cases:
Products billed based on a average metric usage:
For example Hosts (Committed Cost):
Every day if you use a lower number of hosts than committed you will use the committed cost: as soon as you have a bigger number of hosts than committed on demand cost will kick in.
We amortize the cost by spreading the cost over the entire month. If the number of hosts used exceeds the commitment, on-demand costs apply. We can see it on the image under:
Will be transformed into:
2. Products billed based on a cumulative number of:
For example logs Ingested (Usage-Based Committed Cost): Committed usage applies to the first few days of the month, with on-demand costs taking over as usage exceeds the commitment.
Linking Usage with Billing
Datadog provides specific aggregation functions for converting hourly usage into monthly costs for each product. Examples include:
Logs Indexed (30 Days): Aggregated by sum.
Timeseries: Aggregated by the average per hour.
CI Pipeline: Aggregated by the maximum value.
This information helps determine the most accurate way to amortize costs daily to reflect actual usage.
The following table outlines how specific usage metrics correspond to billed product names:
Product Name | Usage Metric |
Logs Ingested | billable_ingested_bytes |
Logs Indexed (X Days) | logs_indexed_events_X_day_count |
APM/Ingested Spans | ingested_spans/ingested_events_bytes |
Custom Events | custom_events_count |
SIEM (Security) | Billed per million events of APM calls |
RUM Lite | RUM units (per thousand sessions) |
Synthetics Browser Checks | browser_check_calls_count |
Synthetic API Tests | synthetics_api/check_calls_count |
API Used:
1. Estimated Cost API
Functionality: Provides a daily updated estimate of the end-of-month (EOM) cost, covering both on-demand and committed usage per product family.
Usage: Every day, the API updates with a more accurate estimate of the final monthly cost. Note that this is an estimate, not a forecast based on partial usage.
2. Historical Data API
Functionality: Offers aggregated cost data from past months, broken down by product family.
Usage: Helps in reviewing historical spending trends and understanding monthly cost distributions.
3. Usage API
Functionality: Delivers hourly usage data for each product family. This includes various metrics, some of which are directly related to billing.
Usage: Allows detailed analysis of usage patterns over time.
4. Usage Attribution (Enterprise Plan Only)
Functionality: Provides detailed attribution of usage costs, helping to break down costs by team, project, or other dimensions.
Reference: Datadog Usage Attribution Documentation
Known Limitation: Datadog Enterprise Accounts Purchased via AWS Marketplace
We’ve identified a limitation affecting customers using Datadog Enterprise accounts purchased through the AWS Marketplace.
Issue
We currently cannot track Datadog spend in Costory if you purchased Datadog enterprise via AWS Marketplace
Our current Datadog integration does not function properly with this account type. This is due to differences in how credentials and API access are managed when Datadog is provisioned via AWS Marketplace on the Datadog side
Coso
Status
We are actively working with the Datadog team to investigate the issue and identify a viable solution.
Workaround
At this time, there is no immediate workaround available. We will update this document and notify affected customers as soon as a tested workaround or permanent fix is ready.