kakakakakku blog

Weekly Tech Blog: Keep on Learning!

AWS CDK で Cost and Usage Reports 2.0 (CUR 2.0) エクスポートを設定する

AWS CDK CUR

AWS CDK で AWS Data Exports の Cost and Usage Reports 2.0 (CUR 2.0) エクスポートを設定する機会があった💰️

L1 コンストラクトの aws_bcmdataexports.CfnExport を設定するときに AWS CloudFormation のドキュメントも確認しながら試行錯誤が必要だった．サンプルコードとして載せておこうと思う📝 ちなみに最初に見たときにモジュール名の bcmdataexports って何？と思うかもしれないけど bcm は AWS Billing And Cost Management のことだと覚えておけば OK👌

docs.aws.amazon.com

👾 lib/cur.ts

import {
    RemovalPolicy,
    Stack,
    StackProps,
    aws_bcmdataexports,
    aws_iam,
    aws_s3,
} from 'aws-cdk-lib'
import { Construct } from 'constructs'

export class CurStack extends Stack {
    constructor(scope: Construct, id: string, props?: StackProps) {
        super(scope, id, props)

        const bucket = new aws_s3.Bucket(this, 'CurBucket', {
            bucketName: 'xxxxx',
            removalPolicy: RemovalPolicy.RETAIN_ON_UPDATE_OR_DELETE,
        })

        bucket.addToResourcePolicy(
            new aws_iam.PolicyStatement({
                actions: [
                    's3:PutObject',
                    's3:GetBucketPolicy'
                ],
                resources: [
                    bucket.bucketArn,
                    `${bucket.bucketArn}/*`
                ],
                principals: [
                    new aws_iam.ServicePrincipal('bcm-data-exports.amazonaws.com'),
                    new aws_iam.ServicePrincipal('billingreports.amazonaws.com'),
                ],
                conditions: {
                    StringLike: {
                        'aws:SourceAccount': `${this.account}`,
                        'aws:SourceArn': [
                            `arn:aws:cur:${this.region}:${this.account}:definition/*`,
                            `arn:aws:bcm-data-exports:${this.region}:${this.account}:export/*`,
                        ]
                    },
                },
            })
        )

        new aws_bcmdataexports.CfnExport(this, 'Cur', {
            export: {
                name: 'kakakakakku-cur',
                destinationConfigurations: {
                    s3Destination: {
                        s3Region: this.region,
                        s3Bucket: bucket.bucketName,
                        s3Prefix: 'cur',
                        s3OutputConfigurations: {
                            format: 'PARQUET',
                            compression: 'PARQUET',
                            outputType: 'CUSTOM',
                            overwrite: 'OVERWRITE_REPORT',
                        },
                    },
                },
                dataQuery: {
                    queryStatement: 'SELECT bill_bill_type, bill_billing_entity, bill_billing_period_end_date, bill_billing_period_start_date, bill_invoice_id, bill_invoicing_entity, bill_payer_account_id, bill_payer_account_name, cost_category, discount, discount_bundled_discount, discount_total_discount, identity_line_item_id, identity_time_interval, line_item_availability_zone, line_item_blended_cost, line_item_blended_rate, line_item_currency_code, line_item_legal_entity, line_item_line_item_description, line_item_line_item_type, line_item_net_unblended_cost, line_item_net_unblended_rate, line_item_normalization_factor, line_item_normalized_usage_amount, line_item_operation, line_item_product_code, line_item_tax_type, line_item_unblended_cost, line_item_unblended_rate, line_item_usage_account_id, line_item_usage_account_name, line_item_usage_amount, line_item_usage_end_date, line_item_usage_start_date, line_item_usage_type, pricing_currency, pricing_lease_contract_length, pricing_offering_class, pricing_public_on_demand_cost, pricing_public_on_demand_rate, pricing_purchase_option, pricing_rate_code, pricing_rate_id, pricing_term, pricing_unit, product, product_comment, product_fee_code, product_fee_description, product_from_location, product_from_location_type, product_from_region_code, product_instance_family, product_instance_type, product_instancesku, product_location, product_location_type, product_operation, product_pricing_unit, product_product_family, product_region_code, product_servicecode, product_sku, product_to_location, product_to_location_type, product_to_region_code, product_usagetype, reservation_amortized_upfront_cost_for_usage, reservation_amortized_upfront_fee_for_billing_period, reservation_availability_zone, reservation_effective_cost, reservation_end_time, reservation_modification_status, reservation_net_amortized_upfront_cost_for_usage, reservation_net_amortized_upfront_fee_for_billing_period, reservation_net_effective_cost, reservation_net_recurring_fee_for_usage, reservation_net_unused_amortized_upfront_fee_for_billing_period, reservation_net_unused_recurring_fee, reservation_net_upfront_value, reservation_normalized_units_per_reservation, reservation_number_of_reservations, reservation_recurring_fee_for_usage, reservation_reservation_a_r_n, reservation_start_time, reservation_subscription_id, reservation_total_reserved_normalized_units, reservation_total_reserved_units, reservation_units_per_reservation, reservation_unused_amortized_upfront_fee_for_billing_period, reservation_unused_normalized_unit_quantity, reservation_unused_quantity, reservation_unused_recurring_fee, reservation_upfront_value, resource_tags, savings_plan_amortized_upfront_commitment_for_billing_period, savings_plan_end_time, savings_plan_instance_type_family, savings_plan_net_amortized_upfront_commitment_for_billing_period, savings_plan_net_recurring_commitment_for_billing_period, savings_plan_net_savings_plan_effective_cost, savings_plan_offering_type, savings_plan_payment_option, savings_plan_purchase_term, savings_plan_recurring_commitment_for_billing_period, savings_plan_region, savings_plan_savings_plan_a_r_n, savings_plan_savings_plan_effective_cost, savings_plan_savings_plan_rate, savings_plan_start_time, savings_plan_total_commitment_to_date, savings_plan_used_commitment, line_item_resource_id FROM COST_AND_USAGE_REPORT',
                    tableConfigurations: {
                        'COST_AND_USAGE_REPORT': {
                            'INCLUDE_RESOURCES': 'TRUE',
                            'INCLUDE_SPLIT_COST_ALLOCATION_DATA': 'FALSE',
                            'TIME_GRANULARITY': 'DAILY',
                        },
                    },
                },
                refreshCadence: {
                    frequency: 'SYNCHRONOUS',
                },
            },
        }).node.addDependency(bucket)
    }
}

デプロイ確認

期待通りに CUR 2.0 エクスポートを設定できている👌

  • ファイルフォーマットは Parquet にする
  • エクスポートするコンテンツとして リソース ID を追加する
  • 粒度は 日次 にする

ポイント

実装のポイントをメモしておこうと思う📝

1. リージョン

CUR 2.0 エクスポートは us-east-1 リージョンに設定する必要がある🌍️

const envVirginia = {
    account: '000000000000',
    region: 'us-east-1',
}

new CurStack(app, 'CurStack', { env: envVirginia })

2. リソース間の依存関係

最初デプロイしようとしたら Amazon S3 バケットポリシーのデプロイ前に CUR 2.0 エクスポートの設定をしようとしてエラーになる場合があった．今回は AWS CDK の .addDependency() を使って Amazon S3 バケットとバケットポリシーのデプロイ後に CUR 2.0 をデプロイするようにリソース間の依存関係を実装した．

AWS CloudFormation で CUR 2.0 をデプロイするサンプルコードは AWS re:Post で紹介されていて，同じように DependsOn が設定されていた👌

CURReportDefinition:
  DependsOn: S3ClientBucketAccessPolicy
  Type: AWS::BCMDataExports::Export

repost.aws

3. Amazon S3 バケットポリシー

Amazon S3 バケットポリシーは一度 CUR 2.0 をマネジメントコンソールで設定して，自動的に設定されるポリシーを参考にした📝

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "bcm-data-exports.amazonaws.com",
                    "billingreports.amazonaws.com"
                ]
            },
            "Action": [
                "s3:GetBucketPolicy",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::xxxxx",
                "arn:aws:s3:::xxxxx/*"
            ],
            "Condition": {
                "StringLike": {
                    "aws:SourceAccount": "000000000000",
                    "aws:SourceArn": [
                        "arn:aws:cur:us-east-1:000000000000:definition/*",
                        "arn:aws:bcm-data-exports:us-east-1:000000000000:export/*"
                    ]
                }
            }
        }
    ]
}

以下のドキュメントに載っている Amazon S3 バケットポリシーは Legacy CUR 専用の設定になっていて，CUR 2.0 だと使えないため注意する😇 CUR 2.0 の Amazon S3 バケットポリシーもドキュメントに載せてくれると助かるんだけどなぁ...

docs.aws.amazon.com

4. S3OutputConfigurations

CUR 2.0 エクスポートの Amazon S3 に関連する設定 S3OutputConfigurations で，今回は Parquet 形式で出力するため format と compression に PARQUET と設定している．overwrite には CREATE_NEW_REPORT と OVERWRITE_REPORT を設定できて，今回は出力ファイルを上書きする OVERWRITE_REPORT を設定した👌

docs.aws.amazon.com

5. dataQuery

dataQuery.queryStatement にはコストデータを取得する SQL クエリを設定する．今回設定した SQL クエリは同じくマネジメントコンソールから取得してそのまま使っている👌運用上不要なカラムがあれば減らすこともできる．あと今回は dataQuery.tableConfigurations に 'INCLUDE_RESOURCES': 'TRUE' を設定して，エクスポートするコンテンツとして リソース ID を追加しているため line_item_resource_id カラムを含めた「計114カラム」になる❗️

SELECT bill_bill_type, bill_billing_entity, bill_billing_period_end_date, bill_billing_period_start_date, bill_invoice_id, bill_invoicing_entity, bill_payer_account_id, bill_payer_account_name, cost_category, discount, discount_bundled_discount, discount_total_discount, identity_line_item_id, identity_time_interval, line_item_availability_zone, line_item_blended_cost, line_item_blended_rate, line_item_currency_code, line_item_legal_entity, line_item_line_item_description, line_item_line_item_type, line_item_net_unblended_cost, line_item_net_unblended_rate, line_item_normalization_factor, line_item_normalized_usage_amount, line_item_operation, line_item_product_code, line_item_tax_type, line_item_unblended_cost, line_item_unblended_rate, line_item_usage_account_id, line_item_usage_account_name, line_item_usage_amount, line_item_usage_end_date, line_item_usage_start_date, line_item_usage_type, pricing_currency, pricing_lease_contract_length, pricing_offering_class, pricing_public_on_demand_cost, pricing_public_on_demand_rate, pricing_purchase_option, pricing_rate_code, pricing_rate_id, pricing_term, pricing_unit, product, product_comment, product_fee_code, product_fee_description, product_from_location, product_from_location_type, product_from_region_code, product_instance_family, product_instance_type, product_instancesku, product_location, product_location_type, product_operation, product_pricing_unit, product_product_family, product_region_code, product_servicecode, product_sku, product_to_location, product_to_location_type, product_to_region_code, product_usagetype, reservation_amortized_upfront_cost_for_usage, reservation_amortized_upfront_fee_for_billing_period, reservation_availability_zone, reservation_effective_cost, reservation_end_time, reservation_modification_status, reservation_net_amortized_upfront_cost_for_usage, reservation_net_amortized_upfront_fee_for_billing_period, reservation_net_effective_cost, reservation_net_recurring_fee_for_usage, reservation_net_unused_amortized_upfront_fee_for_billing_period, reservation_net_unused_recurring_fee, reservation_net_upfront_value, reservation_normalized_units_per_reservation, reservation_number_of_reservations, reservation_recurring_fee_for_usage, reservation_reservation_a_r_n, reservation_start_time, reservation_subscription_id, reservation_total_reserved_normalized_units, reservation_total_reserved_units, reservation_units_per_reservation, reservation_unused_amortized_upfront_fee_for_billing_period, reservation_unused_normalized_unit_quantity, reservation_unused_quantity, reservation_unused_recurring_fee, reservation_upfront_value, resource_tags, savings_plan_amortized_upfront_commitment_for_billing_period, savings_plan_end_time, savings_plan_instance_type_family, savings_plan_net_amortized_upfront_commitment_for_billing_period, savings_plan_net_recurring_commitment_for_billing_period, savings_plan_net_savings_plan_effective_cost, savings_plan_offering_type, savings_plan_payment_option, savings_plan_purchase_term, savings_plan_recurring_commitment_for_billing_period, savings_plan_region, savings_plan_savings_plan_a_r_n, savings_plan_savings_plan_effective_cost, savings_plan_savings_plan_rate, savings_plan_start_time, savings_plan_total_commitment_to_date, savings_plan_used_commitment, line_item_resource_id FROM COST_AND_USAGE_REPORT

ちなみに dataQuery.tableConfigurations に関しては AWS CloudFormation のドキュメントを見ても Type: Object of String 以外の情報がなく困ったけど，同じくマネジメントコンソールの設定を参考にした．今回コストデータの集計単位 TIME_GRANULARITY は日次 DAILY にしてある．

tableConfigurations: {
    'COST_AND_USAGE_REPORT': {
        'INCLUDE_RESOURCES': 'TRUE',
        'INCLUDE_SPLIT_COST_ALLOCATION_DATA': 'FALSE',
        'TIME_GRANULARITY': 'DAILY',
    },
},

docs.aws.amazon.com

まとめ

CUR 2.0 でコスト分析をやっていくぞ〜💪