<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Security on Sam's Blog</title><link>https://blog.samplayskeys.com/categories/security/</link><description>Recent content in Security on Sam's Blog</description><generator>Hugo -- gohugo.io</generator><language>en-us</language><lastBuildDate>Wed, 22 Jul 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://blog.samplayskeys.com/categories/security/index.xml" rel="self" type="application/rss+xml"/><item><title>Securing Your Environment with OpenBao and Tailscale</title><link>https://blog.samplayskeys.com/p/securing-your-environment-with-openbao-and-tailscale/</link><pubDate>Wed, 22 Jul 2026 00:00:00 +0000</pubDate><guid>https://blog.samplayskeys.com/p/securing-your-environment-with-openbao-and-tailscale/</guid><description>&lt;h1 id="securing-your-environment-with-openbao-and-tailscale"&gt;Securing Your Environment with OpenBao and Tailscale&#10;&lt;/h1&gt;&lt;p&gt;&lt;em&gt;By Sam Fleming&lt;/em&gt;&lt;br&gt;&#10;&lt;em&gt;Consultant, Red Hat · Tailscale Insider&lt;/em&gt;&lt;/p&gt;&#10;&lt;hr&gt;&#10;&lt;h2 id="introduction"&gt;Introduction&#10;&lt;/h2&gt;&lt;p&gt;In my day job I&amp;rsquo;m constantly touching enterprise infrastructure, where HashiCorp Vault and OpenBao are standard fixtures. But at home, I run a setup that generates a power bill my electric company sends me personal thank-you notes for.&lt;/p&gt;&#10;&lt;p&gt;At some point I asked myself: &lt;em&gt;How do I bring enterprise-grade security down to my home lab without making my daily workflow a nightmare?&lt;/em&gt;&lt;/p&gt;&#10;&lt;p&gt;That question led me to OpenBao and a pattern for managing secrets that protects self-hosters and engineering teams alike. This post covers the standard container setup, its underlying credential delivery flaw, and how to fix it using OpenBao and Tailscale together.&lt;/p&gt;&#10;&lt;hr&gt;&#10;&lt;h2 id="the-current-environment"&gt;The Current Environment&#10;&lt;/h2&gt;&lt;p&gt;Most self-hosted environments start with containerization. Docker and Podman let us package applications alongside their dependencies and run them in isolation. That ecosystem gives us tools like Paperless-ngx for digitizing mail, Immich for photo management, Nextcloud for file storage, and Forgejo for code hosting. Rather than managing containers individually on the host OS, we define multi-container stacks declaratively using Docker Compose.&lt;/p&gt;&#10;&lt;p&gt;To network these containers securely, Tailscale is my go-to. The cleanest pattern for container networking on a private tailnet is called a &amp;ldquo;sidecar deployment&amp;rdquo;. Instead of exposing ports to the host or giving containers direct LAN access, you run a lightweight Tailscale sidecar container alongside your application in the Compose stack. All incoming and outgoing traffic for that app routes strictly through your private tailnet. You get ACL controls and zero public ports exposed to the internet.&lt;/p&gt;&#10;&lt;p&gt;The catch with this pattern is authentication. Tailscale requires an auth key to register a device on your network, which means that key has to live inside the container deployment.&lt;/p&gt;&#10;&lt;p&gt;In practice, that key usually sits in one of two places:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Plain-text &lt;code&gt;.env&lt;/code&gt; files stored on the host filesystem.&lt;/li&gt;&#10;&lt;li&gt;Hardcoded variables inside CI/CD pipelines (GitHub Actions, GitLab CI, Jenkins).&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;In both cases, you are passing a static, long-lived credential into the environment.&lt;/p&gt;&#10;&lt;p&gt;Most people pull third-party container images from public registries without auditing every upstream dependency. If a dependency in your stack is compromised, an attacker gets shell access inside that container. If your Tailscale auth key sits in an environment variable or a local &lt;code&gt;.env&lt;/code&gt; file, the compromise isn&amp;rsquo;t contained to that single application. An attacker extracts the static key and can spin up malicious nodes from anywhere, attaching them directly to your private tailnet. From there, they can sniff traffic, spoof internal services, and pivot laterally across your network. Features like Tailnet Lock or MAC filtering can help, but the real goal should be making the credential useless if stolen.&lt;/p&gt;&#10;&lt;p&gt;The fix: &lt;em&gt;moving secrets out of flat files and into a dedicated secrets engine.&lt;/em&gt;&lt;/p&gt;&#10;&lt;p&gt;This is where OpenBao, and its predecessor HashiCorp Vault, come in. To understand OpenBao, it helps to look at Vault&amp;rsquo;s history. HashiCorp Vault was the industry standard for secret management and was fully open-source under the Mozilla Public License up through version 1.14. When HashiCorp shifted Vault to a commercial Business Source License in version 1.15, the Linux Foundation launched OpenBao — a community-governed, open-source fork built directly from Vault 1.14. But OpenBao isn&amp;rsquo;t just an encrypted key-value store. It acts as an interactive security broker for authentication, identity, and access control.&lt;/p&gt;&#10;&lt;p&gt;Storing static credentials inside OpenBao is an improvement over &lt;code&gt;.env&lt;/code&gt; files, but it doesn&amp;rsquo;t solve the core problem. If a breached container queries OpenBao for a permanent credential, an attacker can still steal it. However, OpenBao isn&amp;rsquo;t just an encrypted key-value store. It acts as an interactive security broker for authentication, identity, and access control. Instead of holding static secrets, OpenBao brokers access directly with target platforms (AWS, database clusters, networking providers).&lt;/p&gt;&#10;&lt;p&gt;When an application requests access, OpenBao:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Communicates with the provider API.&lt;/li&gt;&#10;&lt;li&gt;Generates a short-lived, single-use credential on the fly.&lt;/li&gt;&#10;&lt;li&gt;Passes the credential to the container with a strict TTL.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;p&gt;If a container is breached after boot, the credential used to start it is already expired. There is no static secret stored in memory to harvest, dropping the blast radius to near zero.&lt;/p&gt;&#10;&lt;hr&gt;&#10;&lt;h2 id="new-day-new-threats"&gt;New Day, New Threats&#10;&lt;/h2&gt;&lt;p&gt;The July 2026 Hugging Face incident is a clear example of why static secrets are a dangerous default.&lt;/p&gt;&#10;&lt;h4 id="what-happened"&gt;What happened&#10;&lt;/h4&gt;&lt;p&gt;On July 16, 2026, Hugging Face disclosed unauthorized access to internal datasets. Five days later, OpenAI confirmed that an autonomous AI agent running inside an evaluation sandbox had escaped its environment, executed 17,000+ actions over a weekend, exploited the data-processing pipeline, and harvested internal service credentials — ultimately accessing cloud infrastructure and internal datasets.&lt;/p&gt;&#10;&lt;p&gt;Public models and Spaces were untouched, and Hugging Face handled the response cleanly (revoke, evict, rebuild). But the post-mortem consensus among security teams was unanimous: the attack method was completely conventional.&lt;/p&gt;&#10;&lt;h4 id="old-playbook-new-attacker"&gt;Old playbook, new attacker&#10;&lt;/h4&gt;&lt;p&gt;As GitGuardian noted in their analysis, if you strip the AI agent from the incident report, the remaining pages read like any breach retrospective from the last decade. The breach didn&amp;rsquo;t rely on exotic zero-days — it succeeded on reusable credentials and flat internal permissions.&lt;/p&gt;&#10;&lt;p&gt;Industry metrics highlight how widespread this vulnerability is:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;Roughly 60.4% of digital identities hold long-lived secrets.&lt;/li&gt;&#10;&lt;li&gt;64% of valid secrets first discovered in 2022 were still active in early 2026.&lt;/li&gt;&#10;&lt;li&gt;Credential-based breaches take an average of 186 days to identify, because a valid credential looks like legitimate traffic.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;h4 id="applying-the-lesson"&gt;Applying the lesson&#10;&lt;/h4&gt;&lt;p&gt;Look at your own environments. Do you have OpenAI or Anthropic API keys sitting in flat config files or &lt;code&gt;.cursor&lt;/code&gt; directories? Tailscale auth keys in Compose &lt;code&gt;.env&lt;/code&gt; files? Database passwords saved in CI/CD pipeline variables? Hugging Face tokens embedded in training scripts?&lt;/p&gt;&#10;&lt;p&gt;Each of those is a standing credential. An attacker — whether a human operator, a ransomware script, or an autonomous model — only needs to find one static key to pivot across your systems.&lt;/p&gt;&#10;&lt;p&gt;Dynamic secrets eliminate that vector. If a Tailscale auth key is minted on demand and expires in minutes, there is nothing permanent to steal.&lt;/p&gt;&#10;&lt;hr&gt;&#10;&lt;h2 id="putting-it-into-practice"&gt;Putting it into practice&#10;&lt;/h2&gt;&lt;p&gt;When evaluating the extent that you want to deploy this, you can tier your approach based on risk.&lt;/p&gt;&#10;&lt;h3 id="tier-1-centralize-static-secrets-in-openbao"&gt;Tier 1: Centralize static secrets in OpenBao&#10;&lt;/h3&gt;&lt;p&gt;Start by getting OpenBao running and moving static secrets out of flat files.&lt;/p&gt;&#10;&lt;p&gt;You can install OpenBao natively (&lt;code&gt;brew install openbao&lt;/code&gt;), run it via official container images on Quay or Docker Hub, or compile it from source.&lt;/p&gt;&#10;&lt;p&gt;For local testing, run it in dev mode:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;bao server -dev&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;p&gt;Writing and fetching keys is straightforward:&lt;/p&gt;&#10;&lt;div class="highlight"&gt;&lt;div class="chroma"&gt;&#10;&lt;table class="lntable"&gt;&lt;tr&gt;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code&gt;&lt;span class="lnt"&gt;1&#10;&lt;/span&gt;&lt;span class="lnt"&gt;2&#10;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&#10;&lt;td class="lntd"&gt;&#10;&lt;pre tabindex="0" class="chroma"&gt;&lt;code class="language-bash" data-lang="bash"&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;bao kv put secret/my-app &lt;span class="nv"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;value&#10;&lt;/span&gt;&lt;/span&gt;&lt;span class="line"&gt;&lt;span class="cl"&gt;bao kv get secret/my-app&#10;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&#10;&lt;/div&gt;&#10;&lt;/div&gt;&lt;p&gt;Even basic centralized secret storage is safer than host &lt;code&gt;.env&lt;/code&gt; files and unencrypted CI variables. Rotate existing static keys, and turn on secret-scanning on your Git repositories to catch accidental commits.&lt;/p&gt;&#10;&lt;h3 id="tier-2-use-dynamic-secrets-for-core-infrastructure"&gt;Tier 2: Use dynamic secrets for core infrastructure&#10;&lt;/h3&gt;&lt;p&gt;For critical workloads, databases, and public-facing services, switch from static to dynamic secrets. Configure OpenBao to mint temporary credentials on demand with short TTLs.&lt;/p&gt;&#10;&lt;h3 id="tier-3-automate-tailscale-keys-with-a-custom-secret-engine"&gt;Tier 3: Automate Tailscale keys with a custom secret engine&#10;&lt;/h3&gt;&lt;p&gt;To solve the Tailscale sidecar issue, I wrote a custom OpenBao Secret Engine plugin for Tailscale in Go.&lt;/p&gt;&#10;&lt;p&gt;Rather than embedding Tailscale API keys into container stacks, you give a single restricted Tailscale OAuth credential to OpenBao. OpenBao registers a dedicated endpoint, such as &lt;code&gt;docker/tailscale/authkey&lt;/code&gt;.&lt;/p&gt;&#10;&lt;p&gt;The startup flow works as follows:&lt;/p&gt;&#10;&lt;ol&gt;&#10;&lt;li&gt;Docker Compose deploys the application stack.&lt;/li&gt;&#10;&lt;li&gt;The Tailscale sidecar boots and requests a key from OpenBao using a short-lived, scoped Vault token.&lt;/li&gt;&#10;&lt;li&gt;OpenBao verifies the token and policy, then calls Tailscale&amp;rsquo;s Control Plane API.&lt;/li&gt;&#10;&lt;li&gt;OpenBao requests a single-use, ephemeral auth key tagged specifically for that service.&lt;/li&gt;&#10;&lt;li&gt;Tailscale returns the key to OpenBao.&lt;/li&gt;&#10;&lt;li&gt;OpenBao passes the key into memory for the sidecar to join the tailnet, then discards it immediately.&lt;/li&gt;&#10;&lt;/ol&gt;&#10;&lt;p&gt;The key never touches disk, is never exposed in logs, and cannot be reused. Once the sidecar authenticates to the tailnet, the key no longer exists.&lt;/p&gt;&#10;&lt;h3 id="expanding-the-pattern"&gt;Expanding the pattern&#10;&lt;/h3&gt;&lt;p&gt;Once OpenBao is brokering access, you can apply the same pattern elsewhere:&lt;/p&gt;&#10;&lt;ul&gt;&#10;&lt;li&gt;&lt;strong&gt;AI &amp;amp; LLM API keys:&lt;/strong&gt; Proxy API calls to OpenAI, Anthropic, or local models through OpenBao instead of storing keys in local config files or &lt;code&gt;.cursor&lt;/code&gt; settings.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;TLS Certificates:&lt;/strong&gt; Issue and renew short-lived TLS certs automatically using cert-manager or Certbot integrations.&lt;/li&gt;&#10;&lt;li&gt;&lt;strong&gt;Identity &amp;amp; Access:&lt;/strong&gt; Use OpenBao as an OIDC broker requiring hardware passkeys or TOTP before granting temporary access.&lt;/li&gt;&#10;&lt;/ul&gt;&#10;&lt;hr&gt;&#10;&lt;h2 id="closing-thoughts"&gt;Closing Thoughts&#10;&lt;/h2&gt;&lt;p&gt;The Hugging Face incident generated headlines because an AI agent executed it. But the underlying issue wasn&amp;rsquo;t novel — it was standing credentials and flat access.&lt;/p&gt;&#10;&lt;p&gt;Moving secrets out of plain-text files and adopting dynamic credentials where it matters limits what an attacker can do, regardless of who or what that attacker happens to be.&lt;/p&gt;&#10;&lt;p&gt;However, the most important thing to rememeber when implementing this is:&lt;/p&gt;&#10;&#10; &lt;blockquote&gt;&#10; &lt;p&gt;&lt;strong&gt;The best security architecture is the one you can actually maintain.&lt;/strong&gt;&lt;/p&gt;&#10;&#10; &lt;/blockquote&gt;&#10;&lt;p&gt;I don&amp;rsquo;t run a complex dynamic pipeline for every single container in my lab. If an architecture is so tedious that you bypass it when you&amp;rsquo;re tired on a Sunday night, it offers zero real protection. Go with what you know, try your best, and be safe.&lt;/p&gt;&#10;&lt;hr&gt;&#10;&lt;p&gt;&lt;em&gt;Slides, code examples, and the OpenBao Tailscale plugin are available on GitHub at &lt;code&gt;@SamPlaysKeys&lt;/code&gt;. You can reach me directly at &lt;code&gt;info@samplayskeys.com&lt;/code&gt;.&lt;/em&gt;&lt;/p&gt;&#10;&lt;p&gt;&lt;em&gt;This post is based on a presentation prepared for SouthEast Linux Fest 2026. Incident details reflect public disclosures as of late July 2026.&lt;/em&gt;&lt;/p&gt;&#10;</description></item></channel></rss>