https://www.dhaval-shah.com/categories/aws/Recent content in AWS on Dhaval ShahHugo -- gohugo.ioSat, 23 May 2026 02:00:50 +0000https://www.dhaval-shah.com/cross-cloud-data-highway-observability/Sat, 23 May 2026 02:00:50 +0000https://www.dhaval-shah.com/cross-cloud-data-highway-observability/Background In the previous article, we explored resiliency and fault tolerance as critical enablers of high‑volume inter‑cloud data transfer. Yet even the most resilient system cannot succeed without deep visibility into its live operations. This final part of the series focuses on observability Observability - A First-Class Requirement RICS builds observability into architecture with two pillars: Metrics Structured logs Pillar-1 : Metrics Throughput Metrics files_transferred_total - counter, labelled by status (SUCCESS, FAILED, DEAD_LETTER) bytes_transferred_total - gauge of cumulative bytes moved _transfer_throughput_mbps - derived metric transfer_backlog_depth - files in PENDING state Latency Metrics chunk_transfer_duration_seconds file_transfer_duration_seconds s3_read_latency_seconds azure_write_latency_seconds Error & Reliability Metrics chunk_retry_count - how many chunks needed retries throttling_events_total - HTTP 429/503 from S3 and Azure dead_letter_files_total - files that exhausted retries commit_failure_total - blob commit failures Concurrency Metrics active_concurrent_chunks - real time in-flight chunk count concurrency_adjutment_events - adaptive controller decisions Pillar-2 : Structured Logs RICS uses JSON-structured logging correlated by transferId.https://www.dhaval-shah.com/cross-cloud-data-highway-high-resiliency/Sat, 16 May 2026 02:00:50 +0000https://www.dhaval-shah.com/cross-cloud-data-highway-high-resiliency/Background Parts 1 and 2 established the what and the how fast — a stateless, chunk-driven pipeline with deterministic block IDs, reactive backpressure, and an adaptive concurrency controller functions under real cloud conditions. But speed and elegant architecture count for nothing if application crash at chunk 73 of 120, or if a transient Azure 503 silently corrupts a partially assembled blob. Part 3 is about the third dimension: how well does RICS survive when things go wrong?https://www.dhaval-shah.com/cross-cloud-data-highway-high-throughput/Mon, 11 May 2026 02:00:50 +0000https://www.dhaval-shah.com/cross-cloud-data-highway-high-throughput/Background In Part 1 we established RICS's foundational architecture — a stateless, chunk-driven pipeline with a hard separation between the data plane and the metadata-driven control plane. That foundation deliberately deferred one critical question: Once the architecture is right, how do you make it fast? Part 2 answers that. We dissect the two mechanisms that directly govern throughput — S3 ranged reads and Azure block uploads — and then examine how chunk sizing, bounded parallelism, and a closed-loop concurrency controller work together to keep the pipeline moving efficiently under real production conditions.https://www.dhaval-shah.com/cross-cloud-data-highway-architecture/Wed, 06 May 2026 02:00:50 +0000https://www.dhaval-shah.com/cross-cloud-data-highway-architecture/Background Imagine you need to move terabytes or petabytes of large files from AWS S3 to Azure Blob Storage, and that too with Reliability & High Throughput The naïve approach (download the whole file, upload whole file) collapses under real‑world realities: Network glitches Process crashes Cloud throttling Feasibility of buffering a 1 TB of file in memory What looks like a simple file-copy operation - is a classic distributed systems problem with: