// case study
Investigating a Production Failure Caused by Excessive Database Calls
A weekend on-call incident where new deployment code triggered an unexpected database overload under production load. I identified the affected data center in Splunk, immediately rerouted traffic to stop the failures, then investigated deep into logs before coordinating a rollback.
Role & Context
Role
Senior Site Reliability Engineer
Company
Visa
On-call
Weekend rotation
As a Senior SRE at Visa, I carried on-call responsibilities for production services. This incident occurred during a weekend rotation, days after a deployment had been made to production during the week.
Problem
A new version of the application had been deployed to production on a weekday without visible issues. Several days later, as weekend traffic patterns increased load on the system, API failures began appearing. The behavior had not surfaced in the development environment — the problem was latent under normal weekday load and only became apparent when request volume grew.\n\nThe root cause, identified later, was that certain API calls in the new code were generating a disproportionately large number of database calls. Under sufficient load, this exhausted database capacity and caused requests to fail.
My Investigation
When the alerts fired, my first action was to check Splunk to determine where the failures were concentrated. The dashboard showed API failure rates rising specifically in the data center where the recent deployment had rolled out — the pattern was geographically scoped, which immediately told me this was not a platform-wide infrastructure event.\n\nWith the affected data center identified, I immediately rerouted production traffic to a healthy data center. This stopped the stream of failed transactions and brought the failure rate down within minutes. My prompt action limited user impact while keeping the investigation options open. Containing the blast radius before the root cause is fully understood is the right call — it protects users and removes time pressure from what comes next.\n\nWith traffic stabilised on the healthy data center, I turned to a thorough investigation. I worked through Splunk logs in depth, then system-level logs, then application logs — checking for infrastructure causes, resource exhaustion, host issues, and network problems before focusing on the application layer. The application logs confirmed failures were concentrated in specific API paths introduced by the recent deployment.\n\nAt that point I engaged the AppDynamics team, who had agents installed on the production servers. Their instrumentation gave visibility into application internals that the standard observability stack could not — specifically, the number of database calls generated per API request. That data revealed the root cause.
Incident Timeline
Incident timeline — reconstructed from memory
New code deployed
Deployment to production data center
Load increases over weekend
Higher request volume surfaces the latent behavior
API failures appear
Alerts fire — on-call engineer paged
Splunk checked — affected data center identified
Failure rate concentrated in the deployment region
Traffic rerouted to healthy data center
Failure rate drops — failed transactions stop
Splunk, system & application logs examined
Failures traced to specific API paths in new deployment
AppDynamics team engaged
Excessive DB calls per API request confirmed as root cause
Rollback deployed & validated
Previous version restored, tested, traffic returned
RCA written & recommendation filed
High-volume dev testing recommended before production deploys
Mitigation & Resolution
Splunk identifies affected data center — traffic immediately rerouted
The first thing I did was check Splunk to establish where failures were occurring. The dashboard showed the issue was concentrated in the data center where the recent deployment had gone out. I immediately rerouted production traffic to a healthy data center. Failure rates dropped within minutes — stopping the stream of failed transactions before the root cause was fully understood.
Deep investigation into Splunk, system, and application logs
With user impact contained, I investigated systematically. I reviewed Splunk logs in depth, then system-level logs, then application logs. The logs ruled out infrastructure causes and confirmed failures were concentrated in specific API paths introduced by the new deployment. I then engaged the AppDynamics team, whose agents on the production servers revealed the root cause: the new code was generating an exceptionally large number of database calls per API request under load — invisible at development-level traffic volumes.
Rollback to previous version
With the deployment confirmed as the cause, I rolled back to the previous production version. The rollback was deployed to the affected data center and validated before traffic was restored to it.
Traffic returned and service confirmed stable
After validating the rollback, traffic was returned to the previously affected data center and monitored to confirm stable operation before closing the incident.
Key Decisions
Shifting traffic before completing root cause analysis
The decision to shift traffic away from the affected data center came before I had a confirmed root cause. The pattern was consistent enough — failures in one region, correlated with a recent deployment — to justify containing the blast radius early. Working the investigation from a position of reduced user impact is preferable to maintaining full exposure while gathering more evidence.
Engaging AppDynamics rather than attempting further self-service investigation
The observability tools I had direct access to — Splunk, system logs, application logs — were sufficient to narrow the problem to the deployment and to the affected region, but not to identify the specific mechanism inside the application. The AppDynamics team had instrumentation visibility I didn't. Recognising that boundary and engaging the right team quickly was the right call rather than continuing to investigate blindly.
Rolling back rather than attempting a hotfix
Once the deployment was confirmed as the cause, rollback was the fastest and safest path to service restoration. A hotfix under incident conditions carries its own risk, and the previous version was known-good. Restoring that state and documenting the problem for a proper fix was the appropriate priority.
Outcomes & Lessons
- →Service was restored by shifting traffic to a healthy data center and rolling back the offending deployment.
- →The incident report I wrote documented the timeline, root cause, and contributing factors — specifically the gap between development-level and production-level database call behavior.
- →I recommended that the development process include testing application code against high call volumes before production deployment, to surface this class of issue earlier in the pipeline.
Lesson
Production load patterns can expose behavior that development environments don't surface. The several-day gap between deployment and visible failure reflects how latent issues accumulate — the code was present from the weekday deploy, but the problem only materialized when weekend traffic applied sufficient pressure. Volume testing in development is not a guarantee, but it reduces the class of surprises that only appear at scale.