Prelude to a Revelation
It was a typical Sunday morning when our team gathered in the conference room, coffee cups in hand and eyes fixed on the whiteboard. We were wrestling with a question that had been nagging us for months: Is secret rotation truly the security blanket everyone believes it to be?
Secret rotation—the practice of regularly changing API keys, tokens, and other credentials—has been a staple in cybersecurity protocols for decades. It’s a checkbox in compliance audits, a line item in security policies, and a task that developers often perform without much enthusiasm. But in an age where cyber threats evolve at lightning speed, we couldn’t shake the feeling that this long-standing practice might be offering a false sense of security.
So, we decided to put it to the test.
Setting the Stage
Our plan was simple on paper but ambitious in scope. We generated various non-human identities—API keys, secrets, tokens, and service account credentials—and “accidentally” leaked them across platforms and environments to observe how quickly these secrets would be discovered and exploited by malicious actors.
During our preparations, the National Institute of Standards and Technology (NIST) released updated guidelines in the second public draft of its Digital Identity Guidelines1 (SP 800-63B) in September 2024. They advised against mandatory periodic password changes unless there is evidence of compromise. This shift resonated with our suspicions and inspired us to delve even deeper. It was as if NIST echoed our suspicions, indicating that the industry is beginning to question traditional security measures.
We brainstormed a list of realistic locations for such leaks, from public GitHub repositories to package managers like NPM and PyPI, and even code sharing websites like Pastebin and JSFiddle. To understand the full scale, we monitored leaks across each platform and carefully tracked every unauthorized access attempt by creating a dedicated key per leak scenario. These were real keys, granting access to real environments. As we were developing our approach, a blog post by Cybenari2 inspired us to dig even deeper, leading us to add some clarifying details to the conversation it started.
Introducing the Series
This blog post serves as a high-level summary of everything we did and what’s to come in our blog series aimed at exposing secret rotation as a security fallacy. The series will unveil how we conducted our research on Non-Human Identities across cloud platforms, code repositories, CI/CD solutions, and SaaS applications.
With AWS re:Invent kicking off this week, where the community gathers to explore the future of cloud security, it’s only fitting that we begin with one of the most common Non-Human Identities in modern cloud environments: AWS Access Keys. Over the coming days, we’ll reveal how quickly these secrets are exploited across various platforms. We’ll also touch on RDS credentials, shedding light on how attackers leverage even the most obscure credentials. In the upcoming weeks, we will share sequential blog posts covering our experiment of leaking NHIs originating in code repositories, CI/CD solutions, and SaaS applications. Spoiler alert: across all terrains, the results should keep you up at night.
The Experiment Unfolds
With our plan in motion, we dispersed the secrets and began the waiting game. What happened next, however, caught us by surprise.
First Contact in Seconds: The quickest exploitation occurred in less than 40 seconds when a leaked secret in a Docker image was detected and used.
GitHub Hotspot: In one case, we leaked a token on a GitHub repository. Within just one minute, attackers had forked our project and initiated unauthorized access. We observed attackers actively scanning Terraform state files, showing that even obscure credentials are fair game.
Platform-Specific Speed: AWS keys leaked on multiple sites showed a notable pattern. Those placed on Docker Hub were accessed almost instantaneously, with GitHub being a close runner-up.
The data painted a clear and alarming picture: automated systems are continuously scanning, and they act fast.
The Harsh Reality
Our findings revealed the limitations of current practices:
Speed of Exploitation: Secrets leaked on highly scanned platforms were compromised in seconds to minutes, with unauthorized activity peaking around early morning UTC.
Illusion of Rotation: Even when we rotated secrets hourly and re-leaked them, new keys were exploited just as quickly. This demonstrated that attackers work faster than most rotation schedules.
Blind Spots in Logging: Many SaaS platforms don’t log critical read operations. For instance, GitHub, Okta, OpenAI, and Twilio lack robust logging for non-privileged actions, meaning a malicious actor could quietly exfiltrate data without a trace. This absence is even more pronounced on platforms like CircleCI, which provides only daily audit logs without IP tracking.
One particularly sobering moment was when a Reddit user commented on a post we created: “Hey, I think you accidentally shared some sensitive keys here.” It was a polite nudge but highlighted just how publicly visible these mistakes can be—and how easily they can be overlooked.
Behind the Curtain: The Attacker’s Perspective
Diving deeper into the exploitation patterns, we observed organized and sophisticated approaches:
Automation at Scale: Attackers employ bots that continuously scrape repositories and forums for exposed secrets, with GitHub and Docker Hub among their most frequently scanned sites.
Global Reach: IP addresses revealed attackers originating from the U.S., Canada, Mexico, and beyond, with China among the most active sources. Some attempts even aimed to access machine-learning models, such as InvokeModel on AWS Bedrock.
Sophisticated Pivoting: Attackers used these secrets to pivot, often escalating privileges or attempting lateral movement, particularly with AWS credentials. This isn’t a casual attacker stumbling upon exposed keys. It’s an automated, organized effort.
A Moment of Clarity
Analyzing the data, the illusion shattered: secret rotation, as practiced today, isn’t the robust security measure it’s believed to be. It’s a Band-Aid on a bullet wound, offering minimal protection against the speed and sophistication of modern cyber threats.
The reality is clear: the window between exposure and rotation leaves sufficient time for attackers to cause significant damage. Rotating secrets after they’ve been compromised is akin to locking the barn door after the horse has bolted.
The Path Forward
Acknowledging the problem is the first step; the next is finding a solution.
Embracing Zero Trust a Ephemeral Secrets
Zero Trust Architecture: Operate on the principle that no user or system is inherently trusted. Every access request is verified, regardless of its origin.
Ephemeral Credentials: Use short-lived, context-aware credentials that expire quickly, minimizing the opportunity for exploitation.
Continuous Monitoring: Implement real-time detection mechanisms to identify and respond to suspicious activities immediately.
By redesigning our security frameworks around these principles, we can create systems that are resilient against the tactics we’ve observed.
Closing Thoughts
Our journey began with a question and led us to a revelation. Secret rotation, while historically significant, is no longer sufficient as a primary defense mechanism. The threats we face today require us to rethink our approaches and adopt strategies that match the speed and cunning of our adversaries.
To every CISO, security practitioner, and developer reading this: it’s time to challenge the status quo. Review your security practices, question long-held beliefs, and embrace new methodologies. The stakes are too high for complacency.
At Clutch, we’re committed to pioneering these new frontiers in cybersecurity. If you’re ready to join us in building a more secure future, we’re here to help.
We invite you to join us over the next few weeks as we embark on a journey to debunk the long-held belief that secret rotation is an effective security strategy. Together, we’ll uncover how attackers exploit secrets at lightning speed, why traditional approaches fall short, and what modern security practices can truly protect your organization.
Over the next few days, we’ll release a new blog post daily, each diving into a different critical area. We’ll begin tomorrow, with a closer look at GitHub and GitLab, uncovering how secrets in these code hosting platforms are exploited almost as soon as they are exposed.
Stay tuned—it’s time to challenge the status quo.
If you can’t wait for the next blog post, we invite you to download the full report, with all scenarios we ran, a deeper dive into our methodologies, platform-specific insights, attacker behavior patterns, and the tool we built to neutralize exposed secrets instantly. It’s an essential read for any organization looking to secure its systems against the escalating risks of exposed secrets.
[1] https://pages.nist.gov/800-63-4/sp800-63b.html
[2] https://cybenari.com/2024/08/whats-the-worst-place-to-leave-your-secrets/