diff --git a/CHANGELOG.md b/CHANGELOG.md index 34a3de8..10616b5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,66 @@ # Change Log -Resources for generating a changelog: +## [Unreleased] + +### Fixed +- **Critical Test Coverage**: Restored `tests/__init__.py` which was accidentally deleted, breaking test discovery and removing all test coverage +- **Input Validation**: Added comprehensive validation for month format inputs in `parse_month_range()` function + - Validates YYYY-MM format with regex pattern matching + - Provides clear error messages for invalid formats + - Handles empty/null inputs gracefully + - Validates month ranges (start cannot be after end) +- **Network Error Handling**: Enhanced network operation error handling in `upload_to_anycost()` function + - Added 30-second timeout for API requests + - Specific handling for timeout, connection, and HTTP errors + - Detailed error messages with request context + - JSON error response parsing for API errors + - Graceful handling of unexpected errors + +### Added +- **AnyCost Stream API Compliance**: Updated `upload_to_anycost()` function to include required `month` parameter in ISO 8601 format (e.g., "2024-08") +- **Batch Processing**: Added support for uploading data to multiple months in a single session + - Single month: `2024-08` + - Month range: `2024-08:2024-10` (uploads to Aug, Sep, Oct) + - Comma-separated: `2024-08,2024-09,2024-11` + - Progress tracking and error resilience for batch uploads +- **Operation Type Support**: Added support for operation types when uploading to AnyCost Stream: + - `replace_drop` (default): Replace all existing data for the month + - `replace_hourly`: Replace data with overlapping hours + - `sum`: Append data to existing records +- **Rich Error Handling**: Comprehensive error handling with helpful messages + - Input validation with retry logic (3 attempts) + - Month format validation with specific error messages + - File processing errors with row-by-row reporting + - Network timeout and connection error handling + - API response validation and error reporting +- **Interactive Prompts**: Added user prompts for processing mode, month selection, and operation type during upload +- **Comprehensive Test Suite**: Added unit tests covering all functions with 20 test cases + - Tests for CSV processing, data transformation, and API upload functionality + - Tests for month range parsing and batch processing functionality + - Mocked external dependencies for reliable testing + - Located in `tests/` directory with pytest framework +- **Developer Experience**: Enhanced documentation and code comments for easy customization + - Step-by-step customization guide for different cloud providers + - Field mapping examples for AWS, Azure, and GCP + - Troubleshooting section with common issues and solutions + - Inline code comments marking customization points + +### Changed +- Enhanced function documentation to explain all required and optional parameters for AnyCost Stream uploads +- Updated file header comments to document month and operation requirements +- Removed beta warning from README as AnyCost Stream is now generally available +- Improved README structure with Quick Start guide and detailed customization instructions + +### Technical Details +- JSON payload now includes `month`, `operation`, and `data` fields as per AnyCost Stream API specification +- Added `parse_month_range()` function to handle different month input formats +- Batch processing makes sequential API calls with error handling and progress tracking +- Maintains backward compatibility while adding new required functionality +- All 20 tests pass successfully with proper mocking of external dependencies + +--- + +## Resources for generating a changelog: [skywinder/Github-Changelog-Generator](https://github.com/skywinder/Github-Changelog-Generator) - generates a full changelog that overwrites the existing CHANGELOG.md. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 37ff6fb..448fe75 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,6 +1,6 @@ # Contribution -Please read [CloudZero contribution guidelines](https://github.com/cloudzero/open-source-template/blob/master/GENERAL-CONTRIBUTING.md). +Please read [CloudZero contribution guidelines](https://github.com/Cloudzero/template-cloudzero-open-source/blob/main/GENERAL-CONTRIBUTING.md). ## Documentation diff --git a/README.md b/README.md index e643304..269ff8f 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,6 @@ This repository contains a Python script that serves as an example of an Adaptor You can use this Adaptor as a model for structuring your own AnyCost Stream Adaptor, modifying it to fit your use case. -**Note:** The AnyCost Stream feature is in beta. Contact your CloudZero representative to request access. ## Table of Contents @@ -64,25 +63,48 @@ An [AnyCost Stream connection](https://docs.cloudzero.com/docs/anycost-stream-ge An [AnyCost Stream Adaptor](https://docs.cloudzero.com/docs/anycost-custom-adaptors) is the code that queries data from the provider, transforms it to fit the required format, and sends the transformed data to CloudZero. +### Quick Start for New Users + +1. **Prerequisites**: Ensure you have Python 3.9+ installed and access to your cost data in CSV format +2. **Setup**: Clone this repository and install dependencies ([Installation](#installation)) +3. **Prepare Data**: Format your CSV files or use the provided examples +4. **Run Script**: Execute with your data files and follow the interactive prompts +5. **Upload**: Choose single month or batch processing to upload to CloudZero + +### Three Core Steps + An AnyCost Stream Adaptor typically performs three actions: 1. [Retrieve data from a cloud provider for a billing month.](#step-1-retrieve-cost-data-from-cloud-provider) 2. [Transform the data into the Common Bill Format (CBF).](#step-2-transform-cost-data-to-cbf) 3. [Send the CBF data to the CloudZero API.](#step-3-send-the-cbf-data-to-cloudzero) -You can write an Adaptor in any language, but this example uses Python. +You can write an Adaptor in any language, but this example uses Python and can be easily customized for different cloud providers. ### Step 1: Retrieve Cost Data From Cloud Provider -Your Adaptor should start by retrieving cost data from your cloud provider. Follow your provider's instructions to retrieve the data you need. For example, this could involve sending requests to the provider's APIs to retrieve billing records for one or more accounts, or downloading a CSV of all cost data from the provider. +Your Adaptor should start by retrieving cost data from your cloud provider. This step varies by provider: -Because every provider makes its cost data available in a different way, the example Adaptor skips this step. Instead, we've provided you with three CSVs representing the data your Adaptor could retrieve from this step: +**Common Data Sources:** +- **AWS**: Cost and Usage Reports (CUR), billing CSV exports +- **Azure**: Cost Management exports, billing data APIs +- **GCP**: Billing export to BigQuery, Cloud Billing API +- **Other Clouds**: Billing APIs, cost management dashboards, CSV exports -- `cloud_usage.csv`: Data related to cloud resource usage -- `cloud_purchase_commitments.csv`: Data for discounts related to committed-use contracts -- `cloud_discounts.csv`: Data for other discounts received +**For This Example:** +Because every provider makes cost data available differently, this example uses three sample CSV files: -The dummy data is taken from the [CBF example](https://docs.cloudzero.com/docs/anycost-common-bill-format-cbf#examples) in the CloudZero documentation. +- `cloud_usage.csv`: Resource usage and compute costs +- `cloud_purchase_commitments.csv`: Reserved instances, savings plans +- `cloud_discounts.csv`: Volume discounts, credits, promotions + +**Customizing for Your Provider:** +To adapt this script for your cloud provider: +1. Replace the CSV reading logic with API calls to your provider +2. Modify the data processing functions to match your provider's data structure +3. Update the column mappings in the transformation functions + +See [Customization Guide](#customizing-for-different-cloud-providers) below for detailed instructions. ### Step 2: Transform Cost Data to CBF @@ -143,6 +165,46 @@ After processing the data, the script will prompt you to upload the CBF data to 1. Enter `y` if you want to upload the data. 2. Provide your AnyCost Stream Connection ID. 3. Enter your CloudZero API key when prompted. +4. Choose processing mode: + - **Single month**: Upload data for one billing month + - **Batch processing**: Upload data for multiple months +5. Specify the billing month(s): + - **Single month**: `2024-08` + - **Month range**: `2024-08:2024-10` (uploads to Aug, Sep, Oct) + - **Comma-separated**: `2024-08,2024-09,2024-11` +6. Choose an operation type: + - **replace_drop** (default): Replace all existing data for the month + - **replace_hourly**: Replace data with overlapping hours + - **sum**: Append data to existing records + +#### Batch Processing Benefits + +- **Time-saving**: Upload historical data for multiple months in one session +- **Progress tracking**: See upload progress and success/failure status for each month +- **Error resilience**: Failed uploads for individual months won't stop the entire process +- **Flexible input**: Support for ranges, lists, or individual months +- **Input validation**: Comprehensive error checking with helpful suggestions +- **Retry logic**: Multiple attempts for invalid input with clear error messages + +#### Error Handling + +The script provides comprehensive error handling and validation: + +**Month Format Validation**: +- Validates YYYY-MM format (e.g., "2024-08") +- Checks for valid date ranges in batch mode +- Provides specific error messages for invalid formats + +**File Processing Errors**: +- Clear messages for missing or inaccessible CSV files +- Validation of required CSV columns +- Row-by-row error reporting with line numbers + +**Network and API Errors**: +- Timeout handling (30-second limit per request) +- Connection error detection +- HTTP status code reporting with error details +- JSON parsing error handling ### Viewing Results @@ -152,6 +214,111 @@ Once uploaded, you can view the processed data within the CloudZero platform. Na To use the `anycost_example.py` script to transform the cost data to CBF, run the command as described in the [Running the Script](#running-the-script) section. +## Testing + +This repository includes a comprehensive test suite to ensure code quality and reliability. + +### Running Tests + +1. Create and activate a virtual environment: + ```bash + python3 -m venv venv + source venv/bin/activate + ``` + +2. Install test dependencies: + ```bash + pip install -r tests/requirements-dev.txt + ``` + +3. Run the test suite: + ```bash + python -m pytest tests/ -v + ``` + +### Test Coverage + +The test suite includes 11 test cases covering: +- CSV reading and processing functions +- Data transformation for usage, commitments, and discounts +- CBF output generation +- AnyCost Stream API upload functionality with mocked requests +- All operation types (replace_drop, replace_hourly, sum) + +All tests use proper mocking to isolate functionality and avoid external dependencies. + +## Customizing for Different Cloud Providers + +This script can be easily adapted for different cloud providers by modifying the data processing functions: + +### Step-by-Step Customization + +1. **Identify Your Data Source** + ```python + # Replace CSV reading with API calls + def get_provider_data(start_date, end_date): + # Example: Call your provider's billing API + # response = provider_client.get_billing_data(start=start_date, end=end_date) + # return response.data + ``` + +2. **Update Data Processing Functions** + ```python + def process_usage_data(raw_data): + # Map your provider's fields to CBF format + cbf_rows = [] + for item in raw_data: + cbf_rows.append({ + "lineitem/type": "Usage", + "resource/service": item["service_name"], # Your field + "resource/id": item["resource_identifier"], # Your field + "time/usage_start": item["billing_period"], # Your field + "cost/cost": str(item["total_cost"]), # Your field + "cost/discounted_cost": str(item["net_cost"]), # Your field + }) + return cbf_rows + ``` + +3. **Common Provider Mappings** + + **AWS CUR Fields:** + - `lineItem/LineItemType` → `lineitem/type` + - `product/ProductName` → `resource/service` + - `lineItem/ResourceId` → `resource/id` + - `lineItem/UsageStartDate` → `time/usage_start` + - `lineItem/UnblendedCost` → `cost/cost` + + **Azure Billing Fields:** + - `MeterCategory` → `resource/service` + - `InstanceId` → `resource/id` + - `UsageDateTime` → `time/usage_start` + - `ExtendedCost` → `cost/cost` + + **GCP Billing Fields:** + - `service.description` → `resource/service` + - `resource.name` → `resource/id` + - `usage_start_time` → `time/usage_start` + - `cost` → `cost/cost` + +4. **Test Your Changes** + ```bash + python -m pytest tests/ -v + ``` + +### Common Troubleshooting + +**Issue: "Missing required columns in CSV"** +- Solution: Update the `required_columns` list in processing functions to match your data + +**Issue: "Invalid cost/discount value"** +- Solution: Check your provider's number format (currency symbols, decimals) + +**Issue: "Invalid month format"** +- Solution: Ensure dates are in YYYY-MM format, convert if needed + +**Issue: "Connection timeout"** +- Solution: Increase timeout in upload function or implement retry logic + ## Contributing We appreciate feedback and contributions to this repo! Before you get started, see [this repo's contribution guide](CONTRIBUTING.md). diff --git a/anycost_example.py b/anycost_example.py index e81e05b..87a88e5 100644 --- a/anycost_example.py +++ b/anycost_example.py @@ -6,6 +6,14 @@ # 1. Query data from a given cloud provider for a billing month # 2. Transform that cloud provider data into Common Billing Format (CBF) # 3. Send that CBF data into the CloudZero platform through an AnyCost Stream connection +# +# When uploading to AnyCost Stream: +# - A billing month (or multiple months) must be specified in ISO 8601 format (e.g., "2024-08") +# - Supports batch processing for multiple months with range or comma-separated formats +# - An operation type can be specified to control how data is handled: +# - replace_drop: Replace all existing data for the month (default) +# - replace_hourly: Replace data with overlapping hours +# - sum: Append data to existing records import csv import decimal @@ -13,6 +21,9 @@ import json import sys import argparse +import re +from datetime import datetime +from typing import List import requests @@ -93,18 +104,191 @@ def write_cbf_rows_to_csv(cbf_rows: list[dict[str, str]], output_file_path: str) writer.writerows(cbf_rows) +def parse_month_range(month_input: str) -> List[str]: + """Parse month input and return list of months. + + Supports: + - Single month: "2024-08" + - Month range: "2024-08:2024-10" (inclusive) + - Comma-separated: "2024-08,2024-09,2024-11" + """ + if not month_input or not month_input.strip(): + raise ValueError("Month input cannot be empty") + + month_pattern = re.compile(r'^\d{4}-\d{2}$') + + if ':' in month_input: + # Handle range format: "2024-08:2024-10" + parts = month_input.split(':') + if len(parts) != 2: + raise ValueError("Month range must have exactly one ':' separator") + + start_str, end_str = parts + start_str, end_str = start_str.strip(), end_str.strip() + + if not month_pattern.match(start_str) or not month_pattern.match(end_str): + raise ValueError("Month format must be YYYY-MM (e.g., '2024-08')") + + try: + start_date = datetime.strptime(start_str + "-01", "%Y-%m-%d") + end_date = datetime.strptime(end_str + "-01", "%Y-%m-%d") + except ValueError as e: + raise ValueError(f"Invalid date format: {e}") + + if start_date > end_date: + raise ValueError("Start month cannot be after end month") + + months = [] + current = start_date + while current <= end_date: + months.append(current.strftime("%Y-%m")) + # Move to next month + if current.month == 12: + current = current.replace(year=current.year + 1, month=1) + else: + current = current.replace(month=current.month + 1) + return months + elif ',' in month_input: + # Handle comma-separated format: "2024-08,2024-09,2024-11" + months = [month.strip() for month in month_input.split(',')] + for month in months: + if not month_pattern.match(month): + raise ValueError(f"Invalid month format '{month}'. Must be YYYY-MM (e.g., '2024-08')") + return months + else: + # Single month + month = month_input.strip() + if not month_pattern.match(month): + raise ValueError(f"Invalid month format '{month}'. Must be YYYY-MM (e.g., '2024-08')") + return [month] + + def upload_to_anycost(cbf_rows: list[dict[str, str]]): - """Upload CBF rows to an AnyCost Stream connection.""" + """Upload CBF rows to an AnyCost Stream connection. + + Supports both single month and batch processing for multiple months. + + Required parameters: + - month(s): Single month, range, or comma-separated list in ISO 8601 format + - Single: "2024-08" + - Range: "2024-08:2024-10" (uploads to Aug, Sep, Oct) + - List: "2024-08,2024-09,2024-11" + - data: List of CBF rows to upload + + Optional parameters: + - operation: How to handle existing data for each month + - "replace_drop" (default): Replace all existing data for the month + - "replace_hourly": Replace data with overlapping hours + - "sum": Append data to existing records + """ anycost_stream_connection_id = input("Enter your AnyCost Stream Connection ID: ") cloudzero_api_key = getpass.getpass("Enter your CloudZero API Key: ") - - response = requests.post( - f"https://api.cloudzero.com/v2/connections/billing/anycost/{anycost_stream_connection_id}/billing_drops", - headers={"Authorization": cloudzero_api_key}, - json={"data": cbf_rows}, - ) - - print(json.dumps(response.json(), indent=2)) + + # Ask user for processing mode + print("\nProcessing mode:") + print("1. Single month") + print("2. Batch processing (multiple months)") + mode_choice = input("Choose processing mode (1-2, default: 1): ").strip() + + if mode_choice == "2": + print("\nBatch processing options:") + print("- Single month: 2024-08") + print("- Month range: 2024-08:2024-10 (inclusive)") + print("- Comma-separated: 2024-08,2024-09,2024-11") + month_input = input("Enter month(s): ") + months = parse_month_range(month_input) + print(f"\nWill process {len(months)} months: {', '.join(months)}") + else: + # Single month mode + month_input = input("Enter the billing month (YYYY-MM format, e.g., 2024-08): ") + months = [month_input] + + # Get the operation type from user + print("\nOperation types:") + print("1. replace_drop (default) - Replace all existing data for the month") + print("2. replace_hourly - Replace data with overlapping hours") + print("3. sum - Append data to existing records") + operation_choice = input("Enter operation type (1-3, default: 1): ").strip() + + operation_map = { + "1": "replace_drop", + "2": "replace_hourly", + "3": "sum", + "": "replace_drop" # default + } + operation = operation_map.get(operation_choice, "replace_drop") + + # Validate months before processing + try: + for month in months: + parse_month_range(month) # Validate each month format + except ValueError as e: + print(f"✗ Invalid month format: {e}") + return + + # Process each month + successful_uploads = 0 + failed_uploads = 0 + + for i, month in enumerate(months, 1): + print(f"\n[{i}/{len(months)}] Uploading data for {month}...") + + try: + response = requests.post( + f"https://api.cloudzero.com/v2/connections/billing/anycost/{anycost_stream_connection_id}/billing_drops", + headers={"Authorization": cloudzero_api_key}, + json={ + "month": month, + "operation": operation, + "data": cbf_rows + }, + timeout=30 + ) + + response.raise_for_status() # Raises HTTPError for bad HTTP status codes + + response_json = response.json() + print(f"Response for {month}:") + print(json.dumps(response_json, indent=2)) + + if response.status_code == 200: + successful_uploads += 1 + print(f"✓ Successfully uploaded data for {month}") + else: + failed_uploads += 1 + print(f"✗ Failed to upload data for {month} (HTTP {response.status_code})") + + except requests.exceptions.Timeout: + failed_uploads += 1 + print(f"✗ Timeout error uploading data for {month}: Request timed out after 30 seconds") + except requests.exceptions.ConnectionError: + failed_uploads += 1 + print(f"✗ Connection error uploading data for {month}: Unable to connect to CloudZero API") + except requests.exceptions.HTTPError as e: + failed_uploads += 1 + print(f"✗ HTTP error uploading data for {month}: {e}") + try: + error_detail = response.json() + print(f"Error details: {json.dumps(error_detail, indent=2)}") + except (ValueError, AttributeError): + pass + except requests.exceptions.RequestException as e: + failed_uploads += 1 + print(f"✗ Network error uploading data for {month}: {e}") + except Exception as e: + failed_uploads += 1 + print(f"✗ Unexpected error uploading data for {month}: {str(e)}") + + # Summary + if len(months) > 1: + print(f"\n=== Batch Upload Summary ===") + print(f"Total months processed: {len(months)}") + print(f"Successful uploads: {successful_uploads}") + print(f"Failed uploads: {failed_uploads}") + if failed_uploads > 0: + print("⚠️ Some uploads failed. Check the error messages above.") + else: + print("✅ All uploads completed successfully!") def main(): diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000..7913242 --- /dev/null +++ b/tests/__init__.py @@ -0,0 +1,5 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2016-2024, CloudZero, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +from .test_anycost_example import * \ No newline at end of file diff --git a/tests/requirements-dev.txt b/tests/requirements-dev.txt new file mode 100644 index 0000000..cf512aa --- /dev/null +++ b/tests/requirements-dev.txt @@ -0,0 +1,2 @@ +pytest>=7.0.0 +requests>=2.25.0 \ No newline at end of file diff --git a/tests/test_anycost_example.py b/tests/test_anycost_example.py new file mode 100644 index 0000000..240b57b --- /dev/null +++ b/tests/test_anycost_example.py @@ -0,0 +1,403 @@ +#!/usr/bin/env python3 +# SPDX-FileCopyrightText: Copyright (c) 2016-2024, CloudZero, Inc. or its affiliates. All Rights Reserved. +# SPDX-License-Identifier: Apache-2.0 + +import csv +import json +import os +import tempfile +import unittest.mock +from unittest.mock import patch, mock_open, MagicMock + +import pytest +import requests + +from anycost_example import ( + read_csv, + process_usage_data, + process_purchase_commitments, + process_discounts, + write_cbf_rows_to_csv, + upload_to_anycost, + parse_month_range, +) + + +class TestReadCsv: + def test_read_csv_success(self): + csv_content = "name,value\ntest,123\nfoo,bar\n" + with patch("builtins.open", mock_open(read_data=csv_content)): + result = read_csv("test.csv") + + expected = [ + {"name": "test", "value": "123"}, + {"name": "foo", "value": "bar"} + ] + assert result == expected + + def test_read_csv_empty_file(self): + csv_content = "name,value\n" + with patch("builtins.open", mock_open(read_data=csv_content)): + result = read_csv("empty.csv") + + assert result == [] + + +class TestProcessUsageData: + def test_process_usage_data_single_row(self): + csv_data = [ + { + "sku": "compute-engine", + "instance_id": "12345", + "usage_date": "2024-08-16T10:00:00Z", + "cost": "100.00", + "discount": "10.00" + } + ] + + with patch("anycost_example.read_csv", return_value=csv_data): + result = process_usage_data("test.csv") + + expected = [{ + "lineitem/type": "Usage", + "resource/service": "compute-engine", + "resource/id": "instance-12345", + "time/usage_start": "2024-08-16T10:00:00Z", + "cost/cost": "100.00", + "cost/discounted_cost": "90.00" + }] + assert result == expected + + def test_process_usage_data_negative_discount(self): + csv_data = [ + { + "sku": "storage", + "instance_id": "67890", + "usage_date": "2024-08-16T11:00:00Z", + "cost": "50.00", + "discount": "-5.00" + } + ] + + with patch("anycost_example.read_csv", return_value=csv_data): + result = process_usage_data("test.csv") + + expected = [{ + "lineitem/type": "Usage", + "resource/service": "storage", + "resource/id": "instance-67890", + "time/usage_start": "2024-08-16T11:00:00Z", + "cost/cost": "50.00", + "cost/discounted_cost": "45.00" + }] + assert result == expected + + +class TestProcessPurchaseCommitments: + def test_process_purchase_commitments(self): + csv_data = [ + { + "commitment_id": "commit-123", + "commitment_date": "2024-08-01T00:00:00Z", + "cost": "1000.00" + } + ] + + with patch("anycost_example.read_csv", return_value=csv_data): + result = process_purchase_commitments("test.csv") + + expected = [{ + "lineitem/type": "CommittedUsePurchase", + "resource/service": "CommittedUse", + "resource/id": "commit-commit-123", + "time/usage_start": "2024-08-01T00:00:00Z", + "cost/cost": "1000.00", + "cost/discounted_cost": "1000.00" + }] + assert result == expected + + +class TestProcessDiscounts: + def test_process_discounts(self): + csv_data = [ + { + "discount_id": "disc-456", + "discount_type": "volume-discount", + "usage_date": "2024-08-16T12:00:00Z", + "discount": "-25.00" + } + ] + + with patch("anycost_example.read_csv", return_value=csv_data): + result = process_discounts("test.csv") + + expected = [{ + "lineitem/type": "Discount", + "resource/service": "volume-discount", + "resource/id": "discount-disc-456", + "time/usage_start": "2024-08-16T12:00:00Z", + "cost/cost": "-25.00", + "cost/discounted_cost": "-25.00" + }] + assert result == expected + + +class TestWriteCbfRowsToCsv: + def test_write_cbf_rows_to_csv(self): + cbf_rows = [ + { + "lineitem/type": "Usage", + "resource/service": "compute", + "resource/id": "instance-123", + "time/usage_start": "2024-08-16T10:00:00Z", + "cost/cost": "50.00", + "cost/discounted_cost": "45.00" + } + ] + + with tempfile.NamedTemporaryFile(mode='w', delete=False, suffix='.csv') as temp_file: + temp_path = temp_file.name + + try: + write_cbf_rows_to_csv(cbf_rows, temp_path) + + with open(temp_path, 'r') as f: + reader = csv.DictReader(f) + result = list(reader) + + assert len(result) == 1 + assert result[0] == cbf_rows[0] + + finally: + os.unlink(temp_path) + + +class TestUploadToAnycost: + @patch('anycost_example.input') + @patch('anycost_example.getpass.getpass') + @patch('anycost_example.requests.post') + @patch('builtins.print') + def test_upload_to_anycost_success(self, mock_print, mock_post, mock_getpass, mock_input): + # Setup mocks + mock_input.side_effect = [ + "connection-123", # AnyCost Stream Connection ID + "1", # single month mode + "2024-08", # month + "1" # operation choice (replace_drop) + ] + mock_getpass.return_value = "api-key-456" + + mock_response = MagicMock() + mock_response.json.return_value = {"status": "success", "message": "Data uploaded successfully"} + mock_post.return_value = mock_response + + cbf_rows = [{"lineitem/type": "Usage", "cost/cost": "10.00"}] + + # Call function + upload_to_anycost(cbf_rows) + + # Verify API call + mock_post.assert_called_once_with( + "https://api.cloudzero.com/v2/connections/billing/anycost/connection-123/billing_drops", + headers={"Authorization": "api-key-456"}, + json={ + "month": "2024-08", + "operation": "replace_drop", + "data": cbf_rows + }, + timeout=30 + ) + + # Verify response was printed (includes processing mode, operation type, and upload status prints) + assert mock_print.call_count >= 5 + # Check that one of the calls contains the JSON response + print_calls_str = str(mock_print.call_args_list) + assert '"status": "success"' in print_calls_str + + @patch('anycost_example.input') + @patch('anycost_example.getpass.getpass') + @patch('anycost_example.requests.post') + @patch('builtins.print') + def test_upload_to_anycost_replace_hourly(self, mock_print, mock_post, mock_getpass, mock_input): + mock_input.side_effect = [ + "connection-789", + "1", # single month mode + "2024-09", + "2" # replace_hourly + ] + mock_getpass.return_value = "api-key-789" + + mock_response = MagicMock() + mock_response.json.return_value = {"status": "success"} + mock_post.return_value = mock_response + + cbf_rows = [] + upload_to_anycost(cbf_rows) + + args, kwargs = mock_post.call_args + assert kwargs["json"]["operation"] == "replace_hourly" + + @patch('anycost_example.input') + @patch('anycost_example.getpass.getpass') + @patch('anycost_example.requests.post') + @patch('builtins.print') + def test_upload_to_anycost_sum_operation(self, mock_print, mock_post, mock_getpass, mock_input): + mock_input.side_effect = [ + "connection-999", + "1", # single month mode + "2024-10", + "3" # sum + ] + mock_getpass.return_value = "api-key-999" + + mock_response = MagicMock() + mock_response.json.return_value = {"status": "success"} + mock_post.return_value = mock_response + + cbf_rows = [] + upload_to_anycost(cbf_rows) + + args, kwargs = mock_post.call_args + assert kwargs["json"]["operation"] == "sum" + + @patch('anycost_example.input') + @patch('anycost_example.getpass.getpass') + @patch('anycost_example.requests.post') + @patch('builtins.print') + def test_upload_to_anycost_default_operation(self, mock_print, mock_post, mock_getpass, mock_input): + mock_input.side_effect = [ + "connection-default", + "1", # single month mode + "2024-11", + "" # empty string should default to replace_drop + ] + mock_getpass.return_value = "api-key-default" + + mock_response = MagicMock() + mock_response.json.return_value = {"status": "success"} + mock_post.return_value = mock_response + + cbf_rows = [] + upload_to_anycost(cbf_rows) + + args, kwargs = mock_post.call_args + assert kwargs["json"]["operation"] == "replace_drop" + + +class TestParseMonthRange: + def test_parse_single_month(self): + result = parse_month_range("2024-08") + assert result == ["2024-08"] + + def test_parse_month_range(self): + result = parse_month_range("2024-08:2024-10") + assert result == ["2024-08", "2024-09", "2024-10"] + + def test_parse_month_range_across_year(self): + result = parse_month_range("2024-11:2025-02") + assert result == ["2024-11", "2024-12", "2025-01", "2025-02"] + + def test_parse_comma_separated_months(self): + result = parse_month_range("2024-08,2024-09,2024-11") + assert result == ["2024-08", "2024-09", "2024-11"] + + def test_parse_comma_separated_with_spaces(self): + result = parse_month_range("2024-08, 2024-09 , 2024-11") + assert result == ["2024-08", "2024-09", "2024-11"] + + def test_parse_single_month_range(self): + result = parse_month_range("2024-08:2024-08") + assert result == ["2024-08"] + + +class TestUploadToAnycostBatch: + @patch('anycost_example.input') + @patch('anycost_example.getpass.getpass') + @patch('anycost_example.requests.post') + @patch('builtins.print') + def test_batch_upload_success(self, mock_print, mock_post, mock_getpass, mock_input): + # Setup mocks for batch mode + mock_input.side_effect = [ + "connection-123", # AnyCost Stream Connection ID + "2", # batch processing mode + "2024-08,2024-09", # months + "1" # operation choice (replace_drop) + ] + mock_getpass.return_value = "api-key-456" + + mock_response = MagicMock() + mock_response.json.return_value = {"status": "success"} + mock_response.status_code = 200 + mock_post.return_value = mock_response + + cbf_rows = [{"lineitem/type": "Usage", "cost/cost": "10.00"}] + + # Call function + upload_to_anycost(cbf_rows) + + # Verify two API calls were made + assert mock_post.call_count == 2 + + # Check first call + first_call = mock_post.call_args_list[0] + assert first_call[1]["json"]["month"] == "2024-08" + assert first_call[1]["json"]["operation"] == "replace_drop" + assert first_call[1]["json"]["data"] == cbf_rows + + # Check second call + second_call = mock_post.call_args_list[1] + assert second_call[1]["json"]["month"] == "2024-09" + assert second_call[1]["json"]["operation"] == "replace_drop" + assert second_call[1]["json"]["data"] == cbf_rows + + @patch('anycost_example.input') + @patch('anycost_example.getpass.getpass') + @patch('anycost_example.requests.post') + @patch('builtins.print') + def test_batch_upload_range(self, mock_print, mock_post, mock_getpass, mock_input): + mock_input.side_effect = [ + "connection-456", + "2", # batch mode + "2024-08:2024-10", # month range + "1" # replace_drop + ] + mock_getpass.return_value = "api-key-789" + + mock_response = MagicMock() + mock_response.json.return_value = {"status": "success"} + mock_response.status_code = 200 + mock_post.return_value = mock_response + + cbf_rows = [] + upload_to_anycost(cbf_rows) + + # Should make 3 calls for Aug, Sep, Oct + assert mock_post.call_count == 3 + + months = [call[1]["json"]["month"] for call in mock_post.call_args_list] + assert months == ["2024-08", "2024-09", "2024-10"] + + @patch('anycost_example.input') + @patch('anycost_example.getpass.getpass') + @patch('anycost_example.requests.post') + @patch('builtins.print') + def test_single_mode_still_works(self, mock_print, mock_post, mock_getpass, mock_input): + mock_input.side_effect = [ + "connection-789", + "1", # single mode + "2024-08", # single month + "1" # replace_drop + ] + mock_getpass.return_value = "api-key-123" + + mock_response = MagicMock() + mock_response.json.return_value = {"status": "success"} + mock_response.status_code = 200 + mock_post.return_value = mock_response + + cbf_rows = [] + upload_to_anycost(cbf_rows) + + # Should make only 1 call + assert mock_post.call_count == 1 + assert mock_post.call_args[1]["json"]["month"] == "2024-08" \ No newline at end of file