Skip to main content

Set-up documentation AWS x Costory

Integrating AWS data – Onboarding Guide

Grégoire Blond avatar
Written by Grégoire Blond
Updated over a month ago

This document will help you:

  • Create a standard data export in the AWS Console

  • Request a backfill from the AWS support team

  • Share your billing data with Costory

  • Answer frequently asked questions

You will only need to provide in Costory:

  • Role ARN of federated Identity

  • Both Bucket & CUR 2.0 names

Prerequisites

Step 0: If you have EKS / ECS

0.1 In Cost management preferences: enable split EKS or ECS if needed. And select based on resource_request

0.2: Activate tags

Important: If a tag is not activated, the data will not be tagged in the CUR 2.0 report, and this cannot be added retroactively.

0.3: If you want to split the cost of EKS / ECS: check all aws:eks / ecs are enabled in https://console.aws.amazon.com/billing/home#/tags cost allocation tags screen.

Step 1: Choose your approach:

You can setup AWS on Costory using two approaches:

  • Cloud Formation Template

  • Manual Setup.

Cloud Formation Template:

Open this web page: Click here

  • Before proceeding, change the AWS Management Console's region to N. Virginia (us-east-1):

    • At the top right corner of the AWS Management Console, you'll see the name of the current region. Click on it.

    • A dropdown menu will appear with a list of available regions. Select "US East(N. Virginia)" (us-east-1) from the list.

  • Click on "Create stack".

  • Follow instructions and click submit.

  • Navigate to Outputs: After the stack creation is complete and store the outputs.

Manual Setup

Step a/ Create a Standard Data Export in the AWS Console

  1. Sign in to your AWS console, and create a new Data Export on the parent billing account.

  2. Choose Standard Data Export

  3. Name your report with an indicative name, such as yourcompany-data-export

  4. Choose CUR 2.0

  5. Include resource IDs, Split cost allocation data and :

    1. If you want to split your EKS data: select HOURLY time granularity

    2. Otherwise select daily

4. Below, check the two checkboxes Parquet and override existing data export file.

5. Configure S3 bucket by creating a new bucket. For example, “costory-exports”

Info: the S3 path prefix should not end with /, AWS adds a trailing slash automatically

6. Verify the policy by clicking Save.

7. Give a path prefix and record this as you will be required to enter it into the Costory console.

8. Click Next and then click Review and Complete. The reports should be created after a few hours.

Step b/ Share Access with Costory

Grant Costory Access manually to Your CUR 2.0 Bucket via federated identity:

  1. Create a new policy and choose JSON editor grant access

2. Cut & paste the JSON below

{
"Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Statement1",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::<YOUR_BUCKET_NAME>",
                "arn:aws:s3:::<YOUR_BUCKET_NAME>/*"
            ]
        }
    ]
}

3. Click Next, name it ‘Costory-read-s3

4. Create policy

5. Create a new role as a custom trust policy with the following right:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "accounts.google.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "accounts.google.com:sub": ["110566759229541603476", "107654210203154458570"]
        }
      }
    }
  ]
}

6. Name that role costory-trust-link, this is a role which allows two of our GCP service accounts (110566759229541603476) in costory-production to impersonate this role and rely on all the rights you will give to this role.

7. Add the correct permission to this role:

Step 2. Request a Backfill:

By default AWS will only start the billing export of the current month, if you wish to retrieve your past data you can use the following approaches:

In case a one year backfill is enough:

  1. Log in to the AWS Management Console.

  2. Navigate to the Billing and Cost Management console.

  3. In the left navigation pane, choose Cost allocation tags.

  4. Select the AWS-generated cost allocation tags tab.

  5. In the list, find the tag aws:createdBy and check the box next to it.

  6. Click the Activate button.

  7. Wait for the tag's status to change to Active.

  8. Once active, check the box for aws:createdBy again.

  9. Click the Backfill tags button (usually located at the top of the table).

  10. Choose the date starting when you want to backfill (you can choose up to 12 months with this method).

If you wish to backfill more than 12 months of data:

Send a support request to backfill the CUR 2.0.

Hi AWS Support,
Please backfill our AWS Cost and Usage Report (CUR 2.0) named <CUR V2 NAME> with the last 36 months of billing data for account ID [Your AWS Account ID]. 
Let me know if anything is needed from our end to proceed.
Thanks

Step 3. Create the AWS Billing Datasource on Costory

You can only setup the AWS account on Costory after AWS performed the first export.

This can take up to 12 hours.

Once you see data on the bucket you can start the setup.

  • Go to Costory -> Datasources -> Billing

  • Click on Import a new Billing Datasource

  • Enter the required information namely

    • Bucket Name

    • Role ARN

    • Prefix (it has to end up with a trailing / )

    • Enable or not eks split data

    • Enter dates only if requested by Costory Support

  • Click Next

  • Click Import

Data will be available the next day or as soon as the AWS export starts.

Frequently Asked Questions

Q. What format should the CUR 2.0 file be in for optimal integration with Costory?

A. For optimal integration, we recommend using the CUR 2.0 (aka Standard Data Export) file in the Parquet format. The Parquet format is preferred for its efficiency in processing and analytics, especially for large-scale data handling.

Q. Does the CUR 2.0 file need to be located in the master payer account?

A. No, the CUR 2.0 file does not necessarily need to be in the master payer account. The important requirement is that the CUR 2.0 must be comprehensive of all billing data for the master payer to ensure accurate and complete data analysis.

Q. Is it acceptable for the CUR 2.0 file to overwrite itself throughout the month?

A. Yes, it is acceptable for the CUR 2.0 file to overwrite itself throughout the month. This practice allows for up-to-date data analysis as new billing information becomes available.

Q. How long does it usually take for data to appear in the Costory platform?

A. It usually takes about 48 hours for Costory to complete the first fetch of data from AWS. We recommend checking first thing in the morning (10 AM your local time) the next day.

Did this answer your question?