https://www.dhaval-shah.com/Recent content on Dhaval ShahHugo -- gohugo.iohttps://www.dhaval-shah.com/cross-cloud-data-highway-architecture/Wed, 06 May 2026 02:00:50 +0000https://www.dhaval-shah.com/cross-cloud-data-highway-architecture/Background Imagine you need to move terabytes or petabytes of large files from AWS S3 to Azure Blob Storage, and that too with Reliability & High Throughput The naïve approach (download the whole file, upload whole file) collapses under real‑world realities: Network glitches Process crashes Cloud throttling Feasibility of buffering a 1 TB of file in memory What looks like a simple file-copy operation - is a classic distributed systems problem with:https://www.dhaval-shah.com/about/Mon, 06 Apr 2026 00:00:00 +0000https://www.dhaval-shah.com/about/I am a Principal Software Architect with more than 2 decades of experience designing and scaling distributed platforms. Specialized in high-throughput, low-latency backend systems built using Java, reactive microservices, and event-driven architectures. Proven track record architecting resilient platforms processing 1000+ TPS with sub-100ms latency, designing distributed pipelines handling 200K+ user provisioning workflows. Recognized for leading complex production problem resolution, institutionalizing NFRs i.e. Costs Optimization, Performance Engineering, Observability, Resiliency as first class requirements and shaping scalable cloud-native architectures.https://www.dhaval-shah.com/contact/Thu, 02 Apr 2026 00:00:00 +0000https://www.dhaval-shah.com/contact/Let's work together and fix whats' slowing you down Independent consulting architect — 20+ years scaling fintech, SaaS, and payment platforms. If your platform has - High‑stakes architecture choices needing an unbiased expert review Performance bottlenecks directly affecting customer experience and revenue Scalability limits blocking growth and investor confidence Cloud costs outpacing platform adoption Book a free 30-min Problem Diagnosis call → No commitment. Tell me what's your single most important pain point — I'll tell you where I'd look first.https://www.dhaval-shah.com/services/Fri, 06 Mar 2026 00:00:00 +0000https://www.dhaval-shah.com/services/Introduction If you’re here, chances are, something in your product or platform or MVP isn’t moving the business forward the way you expected - New features take longer to ship because the tech stack is hard to manage Systems slow down or break under traffic spikes Cloud spend rises faster than customer growth, with no clear driver Outages or poor performance erode customer trust and revenue Ambitious plans need technology that can scale with the business AI‑native products (LLMs, RAG, generative workflows) hit performance, cost, or reliability ceilings This is especially common in startups and small companies where founders and CEOs are focused on product, market fit, and growth.https://www.dhaval-shah.com/building-bulletproof-adf-pipelines/Thu, 27 Nov 2025 02:00:50 +0000https://www.dhaval-shah.com/building-bulletproof-adf-pipelines/Background As software engineers we spend so much of our time building these beautiful, complex data pipelines. We design them to move mountains of information... but let's be honest, we often cross our fingers and hope they don't break. And when they do? A pipeline that fails without making a peep - especially in a system like Azure Data Factory (ADF) - is a genuine nightmare. Working with Azure Data Factory (ADF), I’ve learned that silent failures are the kind you want to avoid at all costs.https://www.dhaval-shah.com/parallel-processing-virtual-threads-reactor-vs-jdk/Wed, 01 Jan 2025 02:00:50 +0000https://www.dhaval-shah.com/parallel-processing-virtual-threads-reactor-vs-jdk/Background My previous article focussed on comparing solutions for performing parallel execution using Spring Core Reactor and JDK 21. This article will follow my previous article where I will provide comparative analysis of Virtual Threads based execution for Spring Core Reactor and JDK 21 based implementation. Keeping the same use case that we referred to for comparing Spring Core Reactor & JDK based implementation, this article will be focussing on:https://www.dhaval-shah.com/parallel-processing-reactor-vs-jdk/Fri, 13 Dec 2024 02:00:50 +0000https://www.dhaval-shah.com/parallel-processing-reactor-vs-jdk/Background If you've ever had to implement business logic that involves concurrent execution along with heavy IO operations, you'll know it can be challenging! Recently, I had a chance to tackle such a use case and made some interesting discoveries about parallel execution in the Java ecosystem. I'll keep this concise and focus on three main areas: Understanding What part of requirements High level overview of available solutions Comparative analysis of available solutions from software & performance engineering standpoint 1.https://www.dhaval-shah.com/rafael-nadal-leadership-lessons/Sun, 13 Oct 2024 02:00:50 +0000https://www.dhaval-shah.com/rafael-nadal-leadership-lessons/Background For someone like me who's been an ardent fan of Rafael Nadal since 2008, hearing the news of his upcoming retirement absolutely shatters my heart. His departure from the sport feels like losing a constant source of inspiration and joy. The courts will never be the same without his relentless pursuit of chasing down every ball, as if his life depended on that point. This is a truly heartbreaking moment for all the Rafans and tennis loving fraternity!https://www.dhaval-shah.com/optimizing-linux-network-stack/Sun, 01 Sep 2024 02:00:50 +0000https://www.dhaval-shah.com/optimizing-linux-network-stack/Background In today’s world of distributed computing, network is the backbone of high-performance, low-latency enterprise applications. Understanding and optimizing the Linux network stack is crucial for achieving peak system performance. Here’s a breakdown of the key components: Network Components: Hardware: Includes routers, NICs, switches, and more. Software: Encompasses the OS kernel, device drivers, and protocols. Protocols: Kernel-Level Protocols: TCP, UDP, etc. Application-Level Protocols: HTTP, WebSocket etc.https://www.dhaval-shah.com/spring-boot-migration-from-2.x-to-3.x/Tue, 07 May 2024 00:00:50 +0000https://www.dhaval-shah.com/spring-boot-migration-from-2.x-to-3.x/Background Lot of my pet projects have been built using Spring Boot 2.x version. Same might be applicable for all the enterprises and organizations who have been building Microservices based applications for their products / services using Spring Boot. During last year, Spring community made a major version upgrade and thereafter released Spring Boot 3.2 version with bunch of new features. First thing that comes to our mind is Why should I migrate my application from 2.https://www.dhaval-shah.com/my-journey-to-ace-ckad-exam/Fri, 12 Jan 2024 06:00:50 +0000https://www.dhaval-shah.com/my-journey-to-ace-ckad-exam/Background As I concluded my year-end break in 2022, I engaged in a profound professional introspection. The realization that struck me most profoundly was the depth of my understanding of the intricate software systems I've been architecting, maintaining, and optimizing for numerous years. The stark truth was that I had merely scratched the surface of comprehending the infrastructure of the software products I've dedicatedly crafted. This realization prompted me to embark on the journey to earn my CKAD certification.https://www.dhaval-shah.com/performance-improvement-vs-performance-optimization/Fri, 13 Oct 2023 06:00:50 +0000https://www.dhaval-shah.com/performance-improvement-vs-performance-optimization/Background Recently I stumbled upon a tweet on my twitter timeline, that discussed about approaches / strategies that can be adopted to improve performance of enterprise applications. What struck me while going through its responses and various similar posts on Linkedin is - tons of material preaching and teaching of how to improve performance of a system is uni-dimensional. While I completely agree to the patterns / guidelines shared by our vibrant community, what IMO is getting missed out is fundamental understanding of Performance Engineering - Primary goal of Performance Engineering is to improve end user experience by reducing latency / increasing throughput.https://www.dhaval-shah.com/optimizing-kafka-producers-consumers/Sat, 01 Apr 2023 06:00:50 +0000https://www.dhaval-shah.com/optimizing-kafka-producers-consumers/Background In this current era, Distributed Architecture has become de-facto architectural paradigm, which necessitates implementation of loosely coupled Microservices which would talk with each other via REST API Message Oriented Middleware As far as Message Oriented Middleware is concerned, Apache Kafka has become quite ubiquitous in today's world of Distributed Systems. Apache Kafka is a powerful, distributed, replicated messaging service platform that is mainly responsible for storing and sharing data in a scalable, robust and fault tolerant manner.https://www.dhaval-shah.com/linux-tools-4-analyzing-system-performance/Sat, 03 Sep 2022 07:00:50 +0000https://www.dhaval-shah.com/linux-tools-4-analyzing-system-performance/Background In today's contemporary world of enterprise software where massively used applications are expected to scale and run seamlessly at extreme high loads e.g. Scaling Hotstar for 25.3 million users, system performance becomes one of the key tenant of architecting high throughput, low latency applications along with capability of ease in scaling as per business / end consumer needs . System performance is a very broad term as it would encompass entire gambit of computer system i.https://www.dhaval-shah.com/g1-gc-primer/Mon, 04 Apr 2022 07:00:50 +0000https://www.dhaval-shah.com/g1-gc-primer/Background In couple of my previous articles I not only tried helping understand fundamentals of JVM memory management, but also delved deeper to understand how Garbage Collection works along with its optimization. So with this background, I am sure you would be convinced that behavior of garbage collection may have ramification on performance of an application. With newer versions of Java, garbage collection has also evolved from Serial -> Parallel -> CMS -> G1 GC -> Z GC.https://www.dhaval-shah.com/performance-comparison-rsocket-webflux/Sun, 12 Sep 2021 14:53:50 +0000https://www.dhaval-shah.com/performance-comparison-rsocket-webflux/Background In one of my previous post we saw tangible advantages (w.r.t throughput, latency and resource utilization) of refactoring existing Microservice application from imperative to reactive constructs. So an obvious question that comes to an inquisitive mind - Can we apply Reactive principles to the underlying communication layer Answer to above question is - Yes and RSocket is the way to go What is RSocket RSocket is a binary protocol that can perform bi-directional communication via TCP.https://www.dhaval-shah.com/performant-and-optimal-spring-webclient/Mon, 15 Mar 2021 14:53:50 +0000https://www.dhaval-shah.com/performant-and-optimal-spring-webclient/Background In my previous post I tried demonstrating how to implement an optimal and performant REST client using RestTemplate In this article I will be demonstrating similar stuff but by using WebClient. But before we get started, lets try rationalizing Why yet another REST client i.e. WebClient IMO there are 2 compelling reasons - Maintenance mode of RestTemplate NOTE: As of 5.0 this class is in maintenance mode, with only minor requests for changes and bugs to be accepted going forward.https://www.dhaval-shah.com/rest-client-with-desired-nfrs-using-springs-resttemplate/Tue, 19 May 2020 14:53:50 +0000https://www.dhaval-shah.com/rest-client-with-desired-nfrs-using-springs-resttemplate/Background In this contemporary world of enterprise application development, Microservice Architecture has become defacto paradigm. With this new paradigm, an application is going to have myriad set of independent and autonomous (micro)services which will be calling each other. One of the fundamental characteristics of Microservice Architecture is Services must be easily consumable Hence most of the services implemented will be exposing REST APIs. In order to consume these REST APIs, each Microservice application will have to implement a REST client.https://www.dhaval-shah.com/refactoring-from-imperative-to-reactive-implementation/Mon, 27 Jan 2020 21:06:28 +0000https://www.dhaval-shah.com/refactoring-from-imperative-to-reactive-implementation/Background As software industry is embracing the new Microservice Architecture paradigm, myriad applications have been built with Spring Boot framework. By the time organizations have got its early versions of microservice applications in production, industry has found out newer and better avenues for further optimizing microservices, so that systems can be more robust, resilient and responsive a.k.a Reactive Systems (as per Reactive Manifesto ). Thanks to Spring Reactor and Spring Webflux which can help us in building reactive systems using Spring framework.https://www.dhaval-shah.com/2019-a-year-gone-by/Sun, 05 Jan 2020 11:19:28 +0000https://www.dhaval-shah.com/2019-a-year-gone-by/Now that twenty-twenties i.e. 2020 has just commenced, I just thought to reflect upon the bygone year 2019 as a casual and occasional blogger. This in my opinion will definitely help me to introspect, analyse and thereby identify hits and misses, which eventually will lead me to improve upon my shortcomings / limitations - yes I am trying to be self critical here as that’s one of the way to improve oneself.https://www.dhaval-shah.com/understanding-and-optimizing-garbage-collection/Sun, 17 Nov 2019 18:37:16 +0000https://www.dhaval-shah.com/understanding-and-optimizing-garbage-collection/In one of my previous post I elaborated basics of JVM memory and also explained how garbage collection works. In this post we will try to understand various types of GC and how to tune and optimize them with a real world example. GC Taxonomy 1. Minor GC When garbage is collected from Young space it is called Minor GC. Characteristics Triggered when JVM is unable to allocate space for a new object in Eden space.https://www.dhaval-shah.com/chaos-engineering-demonstration-with-working-example/Fri, 02 Aug 2019 17:21:13 +0000https://www.dhaval-shah.com/chaos-engineering-demonstration-with-working-example/Background From first part of blog we gathered understanding about basics of Chaos Engineering. Now we will further deep dive to understand how to perform Chaos Engineering with a working example - which to me is going to be quite interesting. First lets start with understanding basics of working example which will be used to demonstrate following- How to perform chaos engineering within an application How to monitor the behavior of the system What to monitor whilst executing experiments on our system Bird's eye view of demo examplehttps://www.dhaval-shah.com/chaos-engineering-a-quick-primer/Sun, 30 Jun 2019 18:13:30 +0000https://www.dhaval-shah.com/chaos-engineering-a-quick-primer/Human body is vulnerable to lot of diseases. So in order to protect human beings from diseases vaccines have been invented. Vaccines mainly work due to process called 'Hormesis', by which system or organism adapts to harm in order to become stronger. Just as our body is susceptible to diseases and germs, so do our systems in software world. Hence vaccines and vaccination can be considered as an apt analogy for understanding Chaos Engineering.https://www.dhaval-shah.com/agile-india-2019-conference-learning-experience/Sun, 14 Apr 2019 19:42:30 +0000https://www.dhaval-shah.com/agile-india-2019-conference-learning-experience/Personally speaking, I am of the belief that only way to excel (professionally) is to collaborate and learn from experts. Since I am constantly striving to improve professionally as Software Craftsman, I try my best to exercise available options. One such option is attending some good Technology Conferences - as they are the platforms from where the so called experts of industry share their distilled experience to the larger community.https://www.dhaval-shah.com/monitoring-spring-boot-application/Fri, 02 Nov 2018 10:37:46 +0000https://www.dhaval-shah.com/monitoring-spring-boot-application/Spring Boot helps developers to implement enterprise grade applications which can be pushed to production in no time. Once application gets into production and if we strongly believe in vedic philosophy of Karma :), we are bound to experience Murphy's Law. Whatever can go wrong, will go wrong Considering nature of Distributed Architecture, Observability and Monitoring of application and infrastructure is of paramount importance. Thanks to the Spring Boot team, who has shipped Spring Boot Actuator from the very first release.https://www.dhaval-shah.com/bootiful-test-driven-development/Sun, 11 Mar 2018 14:28:47 +0000https://www.dhaval-shah.com/bootiful-test-driven-development/Software engineers have been ardently following Test Driven Development (TDD) as an XP practice for having necessary safety nets. I have even tried covering different schools of TDD with an example in one of my previous posts. Considering recent surge in using Spring Boot for developing Microservice applications, I felt a need to understand and learn how to do TDD whilst implementing Spring Boot application. In order to understand how Spring Boot simplifies the overall process of doing TDD, we will consider a very simple use case -https://www.dhaval-shah.com/understanding-jvm-memory-management/Mon, 23 Oct 2017 04:44:34 +0000https://www.dhaval-shah.com/understanding-jvm-memory-management/Everyone of us as Software Engineers would have experienced memory leaks, OOM errors in our Java/JVM applications? In order to dissect such issues it is extremely important to understand the whats' and hows' of JVM memory and its management. JVM - Memory Management One of the many strengths of the JVM is that it performs automatic memory management. As we all know memory management is the process of allocating objects, determining when those objects are no longer needed; thereby de-allocating the memory used by those objects and making it available for future allocationshttps://www.dhaval-shah.com/microservices-and-consumer-driven-contract-testing-using-pact/Mon, 10 Jul 2017 20:37:52 +0000https://www.dhaval-shah.com/microservices-and-consumer-driven-contract-testing-using-pact/Background As per the current trends, Microservice Architecture has become a common paradigm using which enterprise applications are built. With this paradigm shift, an application is going to have myriad set of independent and autonomous (micro)services. So how does a developer do testing within Microservice Architecture? Answer is very obvious - Create integration tests that invokes microservice under test which will internally call dependent microservices Get all the services under test up and running Start executing integration test which invokes microservice under test With this approach, the entire developer level integration testing will have following disadvantages -https://www.dhaval-shah.com/unraveling-the-magic-behind-spring-boot/Sun, 14 May 2017 18:30:00 +0000https://www.dhaval-shah.com/unraveling-the-magic-behind-spring-boot/Considering the extensive usage of Spring Boot for building Cloud Native Architecture, I embarked on the journey of utilizing it in my reference Cloud Native application. When I ran my first application I was literally flabbergasted with the magic Spring Boot does under the hood, using which it camouflages the complexity and challenges of building enterprise applications. In order to understand capabilities of Spring Boot and thereby have its justifiable usage along with its troubleshooting skills I felt a dire need of demystifying the magic behind it!https://www.dhaval-shah.com/understanding-cloud-native-architecture-with-an-example/Thu, 16 Mar 2017 05:20:05 +0000https://www.dhaval-shah.com/understanding-cloud-native-architecture-with-an-example/Whenever the buzz word 'Cloud Native Architecture' gets to our ears, first thing that comes to our mind is - It sounds too jazzy and latest and would have something to do with technology since it has a word called 'Architecture' within it :). Because of this it creates a perception in the minds of lot of people that 'Cloud Native Architecture' is just about TECHNOLOGY. Unfortunately, this is one of the most common misconception that prevails currently in the industry.https://www.dhaval-shah.com/anatomy-of-test-driven-development-part-2/Mon, 16 Jan 2017 18:14:52 +0000https://www.dhaval-shah.com/anatomy-of-test-driven-development-part-2/This is the concluding blog of 2 part series on 'Anatomy of Test Driven Development'. Continuing from where we left in first part of this series, this will mainly talk about 'How' part of TDD along with its avatars. How to do TDD We start with a Test. To get this to work we need to frame the test. We need to make sure that there is enough code in place to make it compile or to make it to work and not throw up run time error.https://www.dhaval-shah.com/anatomy-of-test-driven-development-part-1/Thu, 05 Jan 2017 19:48:47 +0000https://www.dhaval-shah.com/anatomy-of-test-driven-development-part-1/Since we being one of the most intellectual and so called logical species on this earth, we need proper rationale behind each and every action that we do in our personal and professional life. Hence I thought to pen down a two post series highlighting rationale behind following one off the most important and underrated XP practice of software devlopment i.e. Test Driven Development What is TDD Lets try to understand definition of TDD and the primary motive behind TDD; this was outlined by none other than one of the protagonist of TDD i.https://www.dhaval-shah.com/the-curious-case-of-cglib-in-generating-proxies/Tue, 02 Jun 2015 18:42:08 +0000https://www.dhaval-shah.com/the-curious-case-of-cglib-in-generating-proxies/Recently I was required to identify a memory leak in one of the enterprise application running in production. Fortunately we were able to haveheap dumpsfrom the production environment. After analyzing few heap dumps I was able to trace a uniform pattern; which I felt might be one of the potential root causes for the memory leak. Within all the heap dumps, the same object was holding almost 25 % - 35 % of heap memory.