Tailscale SSH failure was not node reachability

An edge node became unreachable through Tailscale SSH.

The operator attempted to connect to the node:

tailscale ssh operator@100.64.x.x

The connection failed with a tailnet policy denial. At first glance, this looked like a normal SSH permission issue, a Linux user problem, or a network connectivity failure.

However, the node was still reachable over the tailnet. Basic network access was working. The failure only happened when using tailscale ssh.

What Made It Confusing

The Headscale ACL already allowed broad network access for admin users:

{
  "action": "accept",
  "src": ["admin@"],
  "dst": ["*:*"]
}

This rule allowed network-level access to the node.

However, Tailscale SSH does not rely only on normal network ACL rules. It has a separate ssh policy block that must explicitly allow the SSH source, destination, and Linux login user.

That means these two operations go through different permission paths:

ssh operator@100.64.x.x
tailscale ssh operator@100.64.x.x

The node could be reachable over the tailnet while still being denied by Tailscale SSH.

Root Cause

A recent ACL change tightened Tailscale SSH access to tagged edge nodes only. The new SSH ACL looked like this:

"ssh": [
  {
    "action": "accept",
    "src": ["tag:sandbox"],
    "dst": ["tag:edge-agent"],
    "users": ["root"]
  }
]

This rule only allowed Tailscale SSH when all of the following were true:

• source was tag:sandbox
• destination was tag:edge-agent
• login user was root

Some existing edge nodes had been provisioned before this tag-based SSH model was introduced. Those legacy nodes had Tailscale SSH enabled:

tailscale up --ssh

But they did not advertise the required destination tag:

tag:edge-agent

The attempted login user was also a non-root operator account, while the ACL only allowed root.

Failure Chain

1. ACL change introduced tag-based Tailscale SSH access.
2. The new SSH ACL required destination nodes to have tag:edge-agent.
3. Some legacy edge nodes had been created before this tag existed.
4. Those nodes had tailscale up --ssh enabled.
5. They did not advertise tag:edge-agent.
6. The operator attempted to log in as a non-root user.
7. The tailscale ssh request did not match any SSH allow rule.
8. Tailscale denied access with a generic policy error.

Resolution

The immediate fix was to either add a temporary compatibility rule for legacy nodes or migrate those nodes to the new tagging model.

Example compatibility rule:

{
  "action": "accept",
  "src": ["admin@"],
  "dst": ["worker@*", "master@*"],
  "users": ["operator", "root"]
}

Affected nodes could also be migrated to advertise the expected tag:

tailscale up --ssh --advertise-tags=tag:edge-agent

The long-term fix was to ensure that newly provisioned nodes always receive the expected tag during onboarding.

Engineering Lesson

This was not a basic SSH failure. It was a production-readiness gap caused by changing the access model without fully handling existing nodes.

The system had two separate permission layers:

• network ACL controls whether packets can reach the node
• SSH ACL controls whether tailscale ssh can log into the node

A production-ready rollout should have included a compatibility rule for existing untagged nodes, a migration path to retag legacy nodes, a provisioning check for new nodes, and validation for nodes with Tailscale SSH enabled but no matching SSH ACL.

Takeaway

The incident was caused by a tag-based Tailscale SSH ACL rollout that did not account for legacy untagged nodes.

The node was not offline. The network was not broken. The normal ACL was not the problem. The real failure was a mismatch between the new SSH ACL model and the state of previously provisioned nodes.