The RK Times
← All posts gRPC Between Services, Securely: The Parts Tutorials Skip

gRPC Between Services, Securely: The Parts Tutorials Skip

By Romaan · Sep 25, 2026 · 34 min read · 5 views

Most gRPC tutorials stop at the point where two services exchange a message. That is the easy half. The half that costs you a weekend is everything after: a schema that stays compatible as both sides ship independently, a transport that actually verifies who is on the other end, and a server that does not fall over when one caller misbehaves.

The trap worth naming up front is that an encrypted service and an authenticated service look identical from the happy path. TLS is on, the calls succeed, the dashboards are green -- and any workload that can route a packet to your pod can still call every method you expose. You find out which one you built during an incident review, not during testing.

This is a working guide, not an overview. One example runs through all of it: an OrderService called by a checkout service. Every section gives you the real syntax, the real commands, and the specific ways each step goes wrong.

Go carries the deep dives, because its security APIs make you say out loud what other stacks hide behind defaults. Python, Java and Node equivalents are in section 9 -- the APIs differ, the model does not.

1. What gRPC actually gives you, and what it does not

Strip away the marketing and gRPC is three things: HTTP/2 for transport, Protocol Buffers for encoding, and generated stubs so neither side hand-writes a client. What that buys you:

  • A schema that is the contract. The .proto file is checked in, versioned, linted and breaking-change-checked in CI. Clients cannot drift from it because their code is generated from it.
  • Binary framing over one multiplexed connection. No head-of-line blocking per request, no connection pool to size, far less CPU than JSON for chatty internal traffic.
  • First-class deadlines and cancellation that propagate across hops, which is the single most effective defence against cascading failure.
  • Four call shapes: unary, server streaming, client streaming, bidirectional streaming.
  • A real security story built into the transport: TLS and mutual TLS are configured on the channel and the server, not bolted on.

And the things it does not do for you, which is the shorter but more important list:

  • Browsers cannot speak gRPC directly. You need grpc-web or Connect with a proxy. For a public, partner-facing API, plain HTTP/JSON is usually still the right answer.
  • Authorization. gRPC authenticates the channel and hands you the caller's identity. Deciding whether that caller may touch this object is entirely your code.
  • Safety by default. A grpc.NewServer() with no options is plaintext, unauthenticated, accepts 4 MB messages, and has no deadline enforcement. Every hardening step in section 7 is opt-in.

Worth taping to the monitor: gRPC secures the pipe and tells you who is at the far end of it. Every question after that one is your application's to answer.

2. The contract: proto3 syntax you will actually use

Put protos in their own directory (ideally their own repo) so that both sides generate from one source of truth. Here is the full example contract, annotated.

syntax = "proto3";

package acme.orders.v1;               // versioned package -- v2 is a NEW package, not an edit

option go_package   = "github.com/acme/apis/gen/go/acme/orders/v1;ordersv1";
option java_package = "com.acme.orders.v1";
option java_multiple_files = true;

import "google/protobuf/timestamp.proto";
import "google/protobuf/field_mask.proto";
import "buf/validate/validate.proto";   // protovalidate constraints, see section 7.6

service OrderService {
  rpc GetOrder    (GetOrderRequest)    returns (Order);
  rpc CreateOrder (CreateOrderRequest) returns (Order);
  rpc ListOrders  (ListOrdersRequest)  returns (ListOrdersResponse);

  // server streaming: one request, many responses
  rpc WatchOrders (WatchOrdersRequest) returns (stream OrderEvent);

  // client streaming: many requests, one response
  rpc UploadReceipts (stream Receipt) returns (UploadSummary);

  // bidirectional streaming: independent read and write halves
  rpc Sync (stream SyncRequest) returns (stream SyncResponse);
}

message Order {
  string id          = 1;
  string customer_id = 2;

  repeated LineItem items = 3;
  Money    total          = 4;
  Status   status         = 5;

  google.protobuf.Timestamp created_at = 6;
  map<string, string>       labels     = 7;

  reserved 8, 9;                 // numbers of deleted fields -- never reuse them
  reserved "legacy_total_cents"; // names too, for JSON/text-format compatibility

  oneof fulfilment {             // at most one of these is set
    ShippingDetails shipping = 10;
    PickupDetails   pickup   = 11;
  }

  optional string promo_code = 12;  // explicit presence: distinguishes "" from unset
}

message LineItem {
  string sku      = 1;
  uint32 quantity = 2;
  Money  unit_price = 3;
}

message Money {
  string currency_code = 1;  // ISO-4217
  int64  minor_units   = 2;  // never use float/double for money
}

enum Status {
  STATUS_UNSPECIFIED = 0;    // REQUIRED: zero value is the unknown/default case
  STATUS_PENDING     = 1;
  STATUS_PAID        = 2;
  STATUS_SHIPPED     = 3;
  STATUS_CANCELLED   = 4;
}

message GetOrderRequest {
  string id = 1 [(buf.validate.field).string.uuid = true];
}

message CreateOrderRequest {
  string   customer_id = 1 [(buf.validate.field).string.uuid = true];
  repeated LineItem items = 2 [(buf.validate.field).repeated = {min_items: 1, max_items: 100}];
  string   idempotency_key = 3 [(buf.validate.field).string.min_len = 8];
}

message ListOrdersRequest {
  string customer_id = 1;
  int32  page_size   = 2 [(buf.validate.field).int32 = {gte: 1, lte: 200}];
  string page_token  = 3;
  google.protobuf.FieldMask read_mask = 4;
}

message ListOrdersResponse {
  repeated Order orders          = 1;
  string         next_page_token = 2;
}

2.1 The wire-compatibility rules that matter

Field numbers, not names, are the wire format. Internalise these:

  • Numbers 1-15 cost one byte of tag; spend them on fields present in every message. 16-2047 cost two bytes.
  • Never change a field's number or type. Adding a field is safe. Deleting a field is safe only if you reserved its number and name.
  • Never renumber, and never reuse a reserved number. An old peer will happily decode new bytes into the old field and corrupt data silently.
  • Enums must have a zero value meaning "unspecified". Unknown enum values arriving from a newer peer are preserved on the wire but read as the raw number, so always handle the default branch.
  • Unknown fields are preserved through parse/serialise round-trips in proto3 (since 3.5), so a proxy that decodes and re-encodes will not drop fields it does not know about.
  • optional on a scalar gives you explicit presence (HasPromoCode() / order.promo_code is not None). Without it, 0, "" and false are indistinguishable from unset. Use it whenever "unset" is a real state, especially for partial updates.
  • Changing a field between optional and repeated, or in and out of a oneof, is a breaking change even though the number stays the same.

2.2 Naming and layout conventions

  • Directory path must mirror the package: proto/acme/orders/v1/order.proto for package acme.orders.v1.
  • Services are PascalCase and end in Service; RPCs are PascalCase; fields are lower_snake_case; enum values are SCREAMING_SNAKE prefixed with the enum name.
  • Every RPC takes its own FooRequest and returns its own FooResponse (except where a resource message is the natural response, as with GetOrder). Reusing a message across two RPCs guarantees a painful day later when only one of them needs a new field.

3. The toolchain: use buf, not bare protoc

Raw protoc works, but it makes you manage include paths, plugin binaries and vendored well-known types by hand, and it gives you no linting or breaking-change detection. buf is the de-facto standard now.

3.1 Install

# buf
brew install bufbuild/buf/buf          # or: go install github.com/bufbuild/buf/cmd/buf@latest

# Go codegen plugins (only needed if you generate locally rather than remotely)
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

# debugging / load-testing tools
go install github.com/fullstorydev/grpcurl/cmd/grpcurl@latest
go install github.com/fullstorydev/grpcui/cmd/grpcui@latest
go install github.com/bojand/ghz/cmd/ghz@latest

3.2 buf.yaml -- module, lint and breaking-change config

# proto/buf.yaml  (buf CLI v1.32+, config schema v2)
version: v2
modules:
  - path: proto
deps:
  - buf.build/bufbuild/protovalidate
lint:
  use:
    - STANDARD
  except:
    - FIELD_NOT_REQUIRED
breaking:
  use:
    - FILE          # strictest: catches anything that breaks generated code
  ignore_unstable_packages: true

3.3 buf.gen.yaml -- what gets generated where

version: v2
managed:
  enabled: true
  override:
    - file_option: go_package_prefix
      value: github.com/acme/apis/gen/go
plugins:
  - remote: buf.build/protocolbuffers/go:v1.36.5
    out: gen/go
    opt: paths=source_relative
  - remote: buf.build/grpc/go:v1.5.1
    out: gen/go
    opt:
      - paths=source_relative
      - require_unimplemented_servers=true   # keep: forces you to embed the Unimplemented struct
  - remote: buf.build/protocolbuffers/python:v29.3
    out: gen/python
  - remote: buf.build/grpc/python:v1.71.0
    out: gen/python

require_unimplemented_servers=true is a compatibility feature, not a nuisance: embedding UnimplementedOrderServiceServer means adding an RPC to the proto will not break the build of every server that has not implemented it yet, and unimplemented methods return codes.Unimplemented instead of panicking.

3.4 Daily commands

buf format -w                    # canonical formatting, run pre-commit
buf lint                         # naming/style rules from buf.yaml
buf generate                     # write the stubs

# the one that saves you: fail CI if the change breaks the wire contract
buf breaking --against 'https://github.com/acme/apis.git#branch=main,subdir=proto'

# build a descriptor set for tools that need the schema without reflection
buf build -o api.binpb --as-file-descriptor-set

3.5 Wire it into CI

# .github/workflows/protos.yml
name: protos
on: [pull_request]
jobs:
  buf:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: bufbuild/buf-action@v1
        with:
          lint: true
          format: true
          breaking: true
          breaking_against: 'https://github.com/${{ github.repository }}.git#branch=main,subdir=proto'
      - name: generated code is up to date
        run: buf generate && git diff --exit-code

The last step is the one people skip and then regret: it guarantees the checked-in stubs match the protos.

3.6 The bare-protoc equivalent, for reference

protoc \
  -I proto \
  -I "$(buf --version >/dev/null && echo third_party)" \
  --go_out=gen/go --go_opt=paths=source_relative \
  --go-grpc_out=gen/go --go-grpc_opt=paths=source_relative \
  proto/acme/orders/v1/order.proto

4. Server and client skeletons

4.1 Implementing the service

buf generate produces OrderServiceServer (the interface you implement), RegisterOrderServiceServer, and OrderServiceClient.

package orders

import (
	"context"

	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	ordersv1 "github.com/acme/apis/gen/go/acme/orders/v1"
)

type Server struct {
	ordersv1.UnimplementedOrderServiceServer // forward compatibility, keep it embedded
	store Store
}

func (s *Server) GetOrder(ctx context.Context, req *ordersv1.GetOrderRequest) (*ordersv1.Order, error) {
	o, err := s.store.Get(ctx, req.GetId())
	switch {
	case errors.Is(err, ErrNotFound):
		// Deliberately terse: do not echo the caller's input or the DB error back.
		return nil, status.Error(codes.NotFound, "order not found")
	case err != nil:
		s.log.ErrorContext(ctx, "get order", "err", err) // detail goes to logs...
		return nil, status.Error(codes.Internal, "internal error") // ...not to the caller
	}
	return o.Proto(), nil
}

// Server streaming: the generated stream is your only output channel.
func (s *Server) WatchOrders(req *ordersv1.WatchOrdersRequest, stream ordersv1.OrderService_WatchOrdersServer) error {
	events, err := s.store.Subscribe(stream.Context(), req.GetCustomerId())
	if err != nil {
		return status.Error(codes.Internal, "subscribe failed")
	}
	for {
		select {
		case <-stream.Context().Done(): // client went away or deadline fired
			return status.FromContextError(stream.Context().Err()).Err()
		case ev, ok := <-events:
			if !ok {
				return nil // clean end of stream
			}
			if err := stream.Send(ev); err != nil {
				return err // transport error, already a status
			}
		}
	}
}

Always use stream.Context() (not a background context) inside streaming handlers: it is cancelled when the client disconnects, which is how you avoid leaking goroutines for clients that vanished.

4.2 Calling it

Note the API: grpc.NewClient replaces grpc.Dial/grpc.DialContext, which are deprecated as of grpc-go 1.63. NewClient is lazy -- it does not connect until the first RPC -- so there is no WithBlock and no dial timeout to tune.

conn, err := grpc.NewClient(
	"dns:///orders.acme.svc.cluster.local:8443",   // scheme matters: dns:/// enables re-resolution
	grpc.WithTransportCredentials(creds),          // section 5
	grpc.WithPerRPCCredentials(tokenSource),       // section 6.2
	grpc.WithDefaultServiceConfig(serviceConfig),  // section 8
	grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)
if err != nil {
	return err
}
defer conn.Close()

client := ordersv1.NewOrderServiceClient(conn)

ctx, cancel := context.WithTimeout(ctx, 2*time.Second) // ALWAYS set a deadline
defer cancel()

order, err := client.GetOrder(ctx, &ordersv1.GetOrderRequest{Id: id})
if err != nil {
	st, _ := status.FromError(err)
	switch st.Code() {
	case codes.NotFound:
		return nil, ErrNoSuchOrder
	case codes.DeadlineExceeded, codes.Unavailable:
		return nil, fmt.Errorf("orders unavailable: %w", err)
	default:
		return nil, err
	}
}

One ClientConn per target service for the whole process. It is safe for concurrent use and multiplexes over HTTP/2; creating a connection per request destroys the performance you came for.

5. Transport security: mutual TLS end to end

Everything up to here is plumbing you could have got from any tutorial. This section is the one that decides whether you shipped a secure service or just an encrypted one.

The default posture for service-to-service gRPC should be mutual TLS: the client verifies the server's certificate and the server verifies the client's. One-way TLS tells the client it reached the right server but lets anyone who can route a packet to your pod call every method you expose. "It is inside the VPC" is not an authentication mechanism -- it is a bet that no workload in the VPC is ever compromised.

5.1 Getting certificates

Do not hand-roll this in production. The classic failure is a CA someone generated once on a laptop, which expires at 3am eighteen months later while the person who holds the key has left. Use one of:

  • SPIFFE/SPIRE -- issues short-lived SVIDs with a URI SAN identifying the workload. The identity model below assumes this shape.
  • cert-manager on Kubernetes with an internal issuer, mounting tls.crt/tls.key/ca.crt into the pod.
  • HashiCorp Vault PKI, or step-ca for smaller setups.
  • A service mesh (Istio, Linkerd) that terminates mTLS in a sidecar -- see 5.6.

That said, generate them by hand once, locally, so the fields stop being magic. The critical detail: gRPC implementations validate the Subject Alternative Name, not the Common Name. A cert with only a CN will fail verification with a confusing x509: certificate relies on legacy Common Name field error.

# ---- 1. A development CA -------------------------------------------------
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
  -subj "/CN=acme-dev-ca" -out ca.crt

# ---- 2. Server certificate, with SANs and serverAuth EKU -----------------
cat > server.cnf <<'CNF'
[req]
distinguished_name = dn
req_extensions     = v3_req
prompt             = no
[dn]
CN = orders.acme.svc.cluster.local
[v3_req]
basicConstraints = CA:FALSE
keyUsage         = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName   = @alt
[alt]
DNS.1 = orders.acme.svc.cluster.local
DNS.2 = orders
DNS.3 = localhost
IP.1  = 127.0.0.1
CNF

openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -config server.cnf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -sha256 -days 365 -out server.crt -extfile server.cnf -extensions v3_req

# ---- 3. Client certificate: clientAuth EKU + a SPIFFE URI SAN as identity -
cat > client.cnf <<'CNF'
[req]
distinguished_name = dn
req_extensions     = v3_req
prompt             = no
[dn]
CN = checkout
[v3_req]
basicConstraints = CA:FALSE
keyUsage         = critical, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth
subjectAltName   = @alt
[alt]
URI.1 = spiffe://acme.internal/ns/prod/sa/checkout
CNF

openssl genrsa -out client.key 2048
openssl req -new -key client.key -out client.csr -config client.cnf
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
  -sha256 -days 90 -out client.crt -extfile client.cnf -extensions v3_req

# ---- 4. Check what you actually produced ---------------------------------
openssl x509 -in client.crt -noout -text | grep -A2 'Alternative Name'

The step CLI does the same in one line each, which is what you want in a Makefile:

step certificate create acme-dev-ca ca.crt ca.key --profile root-ca --no-password --insecure
step certificate create orders.acme.svc.cluster.local server.crt server.key \
  --profile leaf --ca ca.crt --ca-key ca.key --no-password --insecure \
  --san orders.acme.svc.cluster.local --san localhost --not-after 8760h
step certificate create checkout client.crt client.key \
  --profile leaf --ca ca.crt --ca-key ca.key --no-password --insecure \
  --san spiffe://acme.internal/ns/prod/sa/checkout --not-after 2160h

5.2 Server side: require and verify the client certificate

import (
	"crypto/tls"
	"crypto/x509"
	"errors"
	"os"

	"google.golang.org/grpc/credentials"
)

func serverCreds() (credentials.TransportCredentials, error) {
	cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		return nil, fmt.Errorf("load server keypair: %w", err)
	}
	caPEM, err := os.ReadFile("ca.crt")
	if err != nil {
		return nil, err
	}
	pool := x509.NewCertPool()
	if !pool.AppendCertsFromPEM(caPEM) {
		return nil, errors.New("no CA certificates parsed")
	}

	return credentials.NewTLS(&tls.Config{
		Certificates: []tls.Certificate{cert},
		ClientCAs:    pool,
		ClientAuth:   tls.RequireAndVerifyClientCert, // <- this line IS mutual TLS
		MinVersion:   tls.VersionTLS13,
	}), nil
}

The ClientAuth values are a trap worth memorising. Only the last one is mTLS:

  • tls.NoClientCert -- default; no client cert requested.
  • tls.RequestClientCert / tls.RequireAnyClientCert -- a cert may or must be sent, but is not verified against your CA. Useless for authentication.
  • tls.VerifyClientCertIfGiven -- verified only if presented; an attacker simply presents nothing.
  • tls.RequireAndVerifyClientCert -- required and chained to ClientCAs. This is the one.

Pin MinVersion explicitly. Go's default minimum for servers is TLS 1.2; if every peer is under your control, TLS 1.3 removes a whole class of downgrade and cipher-negotiation questions.

5.3 Client side

func clientCreds() (credentials.TransportCredentials, error) {
	cert, err := tls.LoadX509KeyPair("client.crt", "client.key")
	if err != nil {
		return nil, err
	}
	caPEM, _ := os.ReadFile("ca.crt")
	pool := x509.NewCertPool()
	pool.AppendCertsFromPEM(caPEM)

	return credentials.NewTLS(&tls.Config{
		Certificates: []tls.Certificate{cert},
		RootCAs:      pool,                            // trust ONLY our CA, not the system pool
		ServerName:   "orders.acme.svc.cluster.local", // must match a server SAN
		MinVersion:   tls.VersionTLS13,
	}), nil
}

conn, err := grpc.NewClient("dns:///orders.acme.svc.cluster.local:8443",
	grpc.WithTransportCredentials(creds))

Three ways people accidentally disable all of this:

  • grpc.WithTransportCredentials(insecure.NewCredentials()) -- plaintext. Fine in a bufconn unit test, never anywhere else. Grep for it in CI.
  • tls.Config{InsecureSkipVerify: true} -- accepts any certificate, so any host that can intercept the connection is trusted. If it is there because the SAN does not match, fix the SAN.
  • Setting RootCAs to nil -- falls back to the system trust store, so any public CA can mint a certificate your service will accept. Always pin your internal CA.

5.4 Certificate rotation without restarts

SPIRE issues SVIDs with hour-scale lifetimes; cert-manager rotates on a schedule. tls.LoadX509KeyPair at startup means your process is using an expired cert long before it restarts. Two options.

The portable one -- GetCertificate / GetConfigForClient callbacks re-read on each handshake:

type reloader struct {
	mu   sync.RWMutex
	cert *tls.Certificate
}

func (r *reloader) get(*tls.ClientHelloInfo) (*tls.Certificate, error) {
	r.mu.RLock()
	defer r.mu.RUnlock()
	return r.cert, nil
}

// fsnotify or a ticker calls reload(); handshakes pick up the new cert immediately.
cfg := &tls.Config{
	GetCertificate: rl.get,
	ClientCAs:      pool,
	ClientAuth:     tls.RequireAndVerifyClientCert,
	MinVersion:     tls.VersionTLS13,
}

The batteries-included one -- advancedtls with file-watching providers, which also refreshes the trust roots (needed when the CA itself rotates):

import (
	"google.golang.org/grpc/credentials/tls/certprovider/pemfile"
	"google.golang.org/grpc/security/advancedtls"
)

identity, err := pemfile.NewProvider(pemfile.Options{
	CertFile:        "/var/run/secrets/tls/tls.crt",
	KeyFile:         "/var/run/secrets/tls/tls.key",
	RefreshDuration: 30 * time.Minute,
})
roots, err := pemfile.NewProvider(pemfile.Options{
	RootFile:        "/var/run/secrets/tls/ca.crt",
	RefreshDuration: time.Hour,
})

creds, err := advancedtls.NewServerCredentials(&advancedtls.Options{
	IdentityOptions:   advancedtls.IdentityCertificateOptions{IdentityProvider: identity},
	RootOptions:       advancedtls.RootCertificateOptions{RootProvider: roots},
	RequireClientCert: true,
	VerificationType:  advancedtls.CertVerification,
	MinTLSVersion:     tls.VersionTLS13,
})

advancedtls is an experimental package and its option structs have changed shape between releases -- pin your grpc-go version and read the godoc for the exact one you are on. Pair either approach with MaxConnectionAge (section 7) so long-lived connections are periodically forced to re-handshake with fresh material.

5.5 Proving mTLS is actually on

A configuration that looks like mTLS but is not will never announce itself. The happy path passes, because the happy path exercises the one case you got right. The only way to know is to assert the failures:

# No client cert -> must fail
grpcurl -cacert ca.crt orders.acme.svc:8443 list
# expected: ... tls: certificate required

# Wrong CA -> must fail
grpcurl -cacert other-ca.crt -cert client.crt -key client.key orders.acme.svc:8443 list
# expected: x509: certificate signed by unknown authority

# Correct material -> succeeds
grpcurl -cacert ca.crt -cert client.crt -key client.key orders.acme.svc:8443 list

# Inspect the handshake directly
openssl s_client -connect orders.acme.svc:8443 -showcerts -alpn h2 </dev/null
#   look for: "Acceptable client certificate CA names" and ALPN "h2"

Those first two belong in CI, not in a runbook. A config comment reading # mTLS enabled proves nothing about what the server does when a client shows up without a certificate.

5.6 The service-mesh alternative

If you run Istio or Linkerd, the sidecar can do mTLS for you and your application speaks plaintext gRPC to localhost:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod
spec:
  mtls:
    mode: STRICT          # reject any plaintext, mesh-wide
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: orders-allow-checkout
  namespace: prod
spec:
  selector:
    matchLabels:
      app: orders
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/prod/sa/checkout"]
      to:
        - operation:
            paths: ["/acme.orders.v1.OrderService/CreateOrder",
                    "/acme.orders.v1.OrderService/GetOrder"]

Pick one layer and be explicit about it. Running both application mTLS and mesh mTLS gives you double encryption, a sidecar that cannot see method paths for its own policy, and two certificate lifecycles to debug. The usual compromise is mesh mTLS for transport plus application-level authorization on the caller identity the mesh forwards.

On Google Cloud, ALTS (google.golang.org/grpc/credentials/alts) is a third option that handles workload identity without certificates, but it only works between GCP workloads.

6. Who is calling? Authentication and authorization

mTLS answers "is this peer one of ours". It does not answer "which of ours" unless you go and look, and it never answers "may they do this".

6.1 Extracting the caller's identity from the peer certificate

import (
	"google.golang.org/grpc/credentials"
	"google.golang.org/grpc/peer"
)

func callerID(ctx context.Context) (string, error) {
	p, ok := peer.FromContext(ctx)
	if !ok {
		return "", status.Error(codes.Unauthenticated, "no peer information")
	}
	tlsInfo, ok := p.AuthInfo.(credentials.TLSInfo)
	if !ok {
		return "", status.Error(codes.Unauthenticated, "connection is not TLS")
	}

	// VerifiedChains, NOT PeerCertificates: the former has been validated
	// against ClientCAs, the latter is whatever the peer sent.
	chains := tlsInfo.State.VerifiedChains
	if len(chains) == 0 || len(chains[0]) == 0 {
		return "", status.Error(codes.Unauthenticated, "no verified client certificate")
	}
	leaf := chains[0][0]

	for _, u := range leaf.URIs {
		if u.Scheme == "spiffe" {
			return u.String(), nil // spiffe://acme.internal/ns/prod/sa/checkout
		}
	}
	if len(leaf.DNSNames) > 0 {
		return "dns:" + leaf.DNSNames[0], nil
	}
	return "", status.Error(codes.Unauthenticated, "certificate carries no usable identity")
}

tlsInfo.State.PeerCertificates is the trap here, and it is an easy one to fall into because the name sounds right. That field holds the chain the client presented, before validation -- under VerifyClientCertIfGiven its contents are chosen by the caller. VerifiedChains is the one that survived your CA.

6.2 Per-RPC credentials: tokens in metadata

Certificates identify the workload. When you also need to carry an end user, a tenant, or a short-lived scoped token, that goes in metadata. Implement credentials.PerRPCCredentials so it is attached automatically and refreshed centrally:

type tokenCreds struct{ src oauth2.TokenSource }

func (t tokenCreds) GetRequestMetadata(ctx context.Context, uri ...string) (map[string]string, error) {
	tok, err := t.src.Token() // oauth2 TokenSource caches and refreshes for you
	if err != nil {
		return nil, err
	}
	return map[string]string{"authorization": tok.Type() + " " + tok.AccessToken}, nil
}

// Refuse to ever put a bearer token on a plaintext connection.
func (t tokenCreds) RequireTransportSecurity() bool { return true }

conn, _ := grpc.NewClient(target,
	grpc.WithTransportCredentials(tlsCreds),
	grpc.WithPerRPCCredentials(tokenCreds{src: ts}),
)

Returning false from RequireTransportSecurity is how credentials end up on the wire in the clear. There is no legitimate reason to do it outside a test.

For a single call you can attach metadata directly:

ctx = metadata.AppendToOutgoingContext(ctx,
	"x-request-id", reqID,
	"x-tenant-id", tenant,
)

Four metadata rules that are much cheaper to read now than to debug later:

  • Keys are case-insensitive and lowercased on the wire. Authorization and authorization are the same key; look it up in lower case.
  • Keys ending in -bin carry arbitrary bytes and are base64-encoded by the library. Everything else must be printable ASCII -- a raw byte in a normal header kills the stream.
  • Headers arrive before the handler runs; trailers arrive after. Use grpc.SetHeader/grpc.SendHeader and grpc.SetTrailer from the server side.
  • Metadata is not free and not encrypted beyond TLS. Do not put secrets in keys you also log.

6.3 Authenticating the token on the server

A unary interceptor is the right place -- it runs for every method, so a new RPC is protected by default rather than by remembering.

type principalKey struct{}

func authUnary(v TokenVerifier) grpc.UnaryServerInterceptor {
	return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler) (any, error) {

		if isPublicMethod(info.FullMethod) { // health checks, reflection if enabled
			return handler(ctx, req)
		}

		md, ok := metadata.FromIncomingContext(ctx)
		if !ok {
			return nil, status.Error(codes.Unauthenticated, "missing metadata")
		}
		vals := md.Get("authorization") // already lowercased by the library
		if len(vals) == 0 {
			return nil, status.Error(codes.Unauthenticated, "missing authorization")
		}
		raw, ok := strings.CutPrefix(vals[0], "Bearer ")
		if !ok {
			return nil, status.Error(codes.Unauthenticated, "malformed authorization")
		}

		// Verify signature against the JWKS, and the registered claims.
		// Skipping aud/iss is how a token minted for another service gets accepted here.
		claims, err := v.Verify(ctx, raw, WithAudience("orders.acme.internal"),
			WithIssuer("https://idp.acme.internal/"))
		if err != nil {
			return nil, status.Error(codes.Unauthenticated, "invalid token")
		}

		return handler(context.WithValue(ctx, principalKey{}, claims.Principal()), req)
	}
}

Streaming needs its own interceptor, and because ServerStream.Context() is read-only you wrap it:

type ctxStream struct {
	grpc.ServerStream
	ctx context.Context
}

func (s ctxStream) Context() context.Context { return s.ctx }

func authStream(v TokenVerifier) grpc.StreamServerInterceptor {
	return func(srv any, ss grpc.ServerStream, info *grpc.StreamServerInfo,
		handler grpc.StreamHandler) error {
		ctx, err := authenticate(ss.Context(), v)
		if err != nil {
			return err
		}
		return handler(srv, ctxStream{ServerStream: ss, ctx: ctx})
	}
}

Note for streaming: the token is validated once, at stream open. A bidi stream held for six hours outlives its five-minute token. If that matters, cap stream lifetime with MaxConnectionAge and re-validate periodically inside the handler.

Use codes.Unauthenticated when identity is missing or invalid, and codes.PermissionDenied when a known identity is not allowed. Conflating them makes client retry logic wrong: the first is worth retrying with a fresh token, the second never is.

6.4 Authorization: gRPC's built-in policy engine

For coarse "which service may call which method", grpc-go ships an RBAC engine driven by a JSON policy, so the rules live in config rather than in code:

{
  "name": "orders-service",
  "allow_rules": [
    {
      "name": "checkout-may-write",
      "source": { "principals": ["spiffe://acme.internal/ns/prod/sa/checkout"] },
      "request": {
        "paths": [
          "/acme.orders.v1.OrderService/CreateOrder",
          "/acme.orders.v1.OrderService/GetOrder"
        ]
      }
    },
    {
      "name": "reporting-may-read",
      "source": { "principals": ["spiffe://acme.internal/ns/prod/sa/reporting"] },
      "request": { "paths": ["/acme.orders.v1.OrderService/ListOrders"] }
    }
  ],
  "deny_rules": [
    {
      "name": "no-admin-from-outside",
      "source": { "principals": ["*"] },
      "request": { "paths": ["/acme.orders.v1.AdminService/*"] }
    }
  ]
}
import "google.golang.org/grpc/authz"

// Re-reads the file on an interval, so policy changes need no redeploy.
i, err := authz.NewFileWatcher("/etc/authz/orders.json", 5*time.Minute)
if err != nil {
	return err
}
srv := grpc.NewServer(
	grpc.Creds(creds),
	grpc.ChainUnaryInterceptor(i.UnaryInterceptor),
	grpc.ChainStreamInterceptor(i.StreamInterceptor),
)

Semantics: deny rules are evaluated first, then allow rules, and anything not matched is denied. Principals come from the validated peer certificate, which is why this only means anything with RequireAndVerifyClientCert. authz.NewStatic(policyJSON) is the no-reload variant.

6.5 Authorization the policy engine cannot do

Method-level RBAC does not know that order o_123 belongs to customer c_456. Object-level checks belong in the handler, next to the data:

func (s *Server) GetOrder(ctx context.Context, req *ordersv1.GetOrderRequest) (*ordersv1.Order, error) {
	p := principalFrom(ctx)

	o, err := s.store.Get(ctx, req.GetId())
	if err != nil { /* ... */ }

	if !p.CanRead(o.CustomerID) {
		// Same code and message as "not found": otherwise the error itself
		// tells an attacker which order IDs exist.
		return nil, status.Error(codes.NotFound, "order not found")
	}
	return o.Proto(), nil
}

Two rules: deny by default (a new RPC with no policy entry must fail closed), and do not let error codes leak existence. Returning PermissionDenied for objects that exist and NotFound for ones that do not is a working enumeration oracle.

7. Hardening the server beyond authentication

7.1 The whole server, wired up

This is the centrepiece. Every option below exists because its default is unsafe or unbounded.

func newServer(cfg Config, deps Deps) (*grpc.Server, error) {
	creds, err := serverCreds() // section 5.2
	if err != nil {
		return nil, err
	}

	validator, err := protovalidate.New()
	if err != nil {
		return nil, err
	}

	authzI, err := authz.NewFileWatcher(cfg.AuthzPolicyPath, 5*time.Minute)
	if err != nil {
		return nil, err
	}

	srv := grpc.NewServer(
		grpc.Creds(creds),

		// Order matters: recovery outermost so it catches panics in the rest;
		// authn before authz; validation after authz so unauthorised callers
		// learn nothing about your schema constraints.
		grpc.ChainUnaryInterceptor(
			recoveryUnary(deps.Log),
			loggingUnary(deps.Log),      // redacts metadata, see 7.7
			authUnary(deps.Verifier),
			authzI.UnaryInterceptor,
			deadlineUnary(30*time.Second),
			validateUnary(validator),
		),
		grpc.ChainStreamInterceptor(
			recoveryStream(deps.Log),
			loggingStream(deps.Log),
			authStream(deps.Verifier),
			authzI.StreamInterceptor,
		),

		grpc.StatsHandler(otelgrpc.NewServerHandler()),

		// Resource bounds.
		grpc.MaxRecvMsgSize(4<<20),      // default is 4 MiB; make it explicit and size it to your data
		grpc.MaxSendMsgSize(4<<20),      // default is effectively unlimited -- always cap this
		grpc.MaxConcurrentStreams(256),  // default is unlimited: one client can exhaust the server
		grpc.MaxHeaderListSize(16<<10),
		grpc.ConnectionTimeout(10*time.Second), // cap on the handshake itself

		// Connection lifecycle: bounds blast radius, forces periodic re-handshake
		// (so rotated certs take effect), and lets the LB rebalance.
		grpc.KeepaliveParams(keepalive.ServerParameters{
			MaxConnectionIdle:     5 * time.Minute,
			MaxConnectionAge:      30 * time.Minute,
			MaxConnectionAgeGrace: 5 * time.Minute,
			Time:                  2 * time.Minute,
			Timeout:               20 * time.Second,
		}),
		// Defence against keepalive-ping floods (CVE-2019-9512 class).
		grpc.KeepaliveEnforcementPolicy(keepalive.EnforcementPolicy{
			MinTime:             30 * time.Second,
			PermitWithoutStream: false,
		}),
	)

	ordersv1.RegisterOrderServiceServer(srv, deps.Orders)

	hs := health.NewServer()
	healthpb.RegisterHealthServer(srv, hs)
	hs.SetServingStatus("acme.orders.v1.OrderService", healthpb.HealthCheckResponse_SERVING)

	// Reflection makes grpcurl work without a descriptor set -- and hands an
	// attacker your complete API surface. Gate it on environment.
	if cfg.Env != "production" {
		reflection.Register(srv)
	}

	return srv, nil
}

7.2 Deadlines

A server that trusts the client to set a deadline will eventually meet a client that does not. Enforce a ceiling:

func deadlineUnary(max time.Duration) grpc.UnaryServerInterceptor {
	return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler) (any, error) {
		dl, ok := ctx.Deadline()
		if !ok || time.Until(dl) > max {
			var cancel context.CancelFunc
			ctx, cancel = context.WithTimeout(ctx, max)
			defer cancel()
		}
		return handler(ctx, req)
	}
}

And inside handlers, always derive downstream contexts from the incoming one. Using context.Background() for a call to the database or the next service breaks cancellation propagation, and a client that gave up is then still paying for work nobody will read.

7.3 Panic recovery

An unrecovered panic in a handler kills the process in some stacks and, worse, can return an error message containing a stack trace.

func recoveryUnary(log *slog.Logger) grpc.UnaryServerInterceptor {
	return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler) (resp any, err error) {
		defer func() {
			if r := recover(); r != nil {
				log.ErrorContext(ctx, "panic in handler",
					"method", info.FullMethod, "panic", r, "stack", string(debug.Stack()))
				err = status.Error(codes.Internal, "internal error") // no detail to the caller
			}
		}()
		return handler(ctx, req)
	}
}

github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/recovery provides this, plus ready-made logging, retry, ratelimit and auth interceptors -- prefer it to hand-rolling once you need more than one.

7.4 Rate limiting

func rateLimitUnary(lim func(principal string) *rate.Limiter) grpc.UnaryServerInterceptor {
	return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler) (any, error) {
		if !lim(principalFrom(ctx).ID).Allow() {
			return nil, status.Error(codes.ResourceExhausted, "rate limit exceeded")
		}
		return handler(ctx, req)
	}
}

Limit per authenticated principal, not per connection: with HTTP/2 multiplexing one connection can carry every request a caller makes, so per-connection limits do nothing.

7.5 Message size and streaming

The 4 MiB default receive limit is a security control, not an annoyance. Raising it to accommodate one large payload raises it for every method. Prefer to stream:

// Client streaming handler with an explicit cap on total bytes.
func (s *Server) UploadReceipts(stream ordersv1.OrderService_UploadReceiptsServer) error {
	const maxTotal = 64 << 20
	var total int
	for {
		r, err := stream.Recv()
		if errors.Is(err, io.EOF) {
			return stream.SendAndClose(&ordersv1.UploadSummary{Count: int32(n)})
		}
		if err != nil {
			return err
		}
		if total += len(r.GetBytes()); total > maxTotal {
			return status.Error(codes.ResourceExhausted, "upload too large")
		}
		// ...
	}
}

Per-call overrides exist when one method genuinely differs: grpc.MaxCallRecvMsgSize(n) as a CallOption on the client, grpc.MaxRecvMsgSize server-wide.

7.6 Input validation with protovalidate

Generated code enforces types, not values. protovalidate puts the constraints in the proto (so every language gets the same rules) and evaluates them with CEL at runtime.

import "buf/validate/validate.proto";

message CreateOrderRequest {
  string customer_id = 1 [(buf.validate.field).string.uuid = true];
  repeated LineItem items = 2 [(buf.validate.field).repeated = {min_items: 1, max_items: 100}];
  string idempotency_key = 3 [(buf.validate.field).string = {min_len: 8, max_len: 64}];

  option (buf.validate.message).cel = {
    id: "order.total_positive"
    message: "order must contain at least one billable item"
    expression: "this.items.exists(i, i.quantity > 0)"
  };
}
import "buf.build/go/protovalidate"

func validateUnary(v protovalidate.Validator) grpc.UnaryServerInterceptor {
	return func(ctx context.Context, req any, info *grpc.UnaryServerInfo,
		handler grpc.UnaryHandler) (any, error) {
		if m, ok := req.(proto.Message); ok {
			if err := v.Validate(m); err != nil {
				// Field-level detail is safe here: it is about the caller's own input.
				return nil, status.Error(codes.InvalidArgument, err.Error())
			}
		}
		return handler(ctx, req)
	}
}

This replaces the deprecated protoc-gen-validate. Validation is not authorization: passing constraints says the request is well-formed, not that the caller may make it.

7.7 Errors and logging

// Bad: leaks schema, host names and internals to whoever called you.
return nil, status.Errorf(codes.Internal, "query failed: %v", err)

// Good: caller gets a code and a stable message; you get the detail in logs.
log.ErrorContext(ctx, "query orders", "err", err, "order_id", id)
return nil, status.Error(codes.Internal, "internal error")

When the client genuinely needs structured detail, use google.rpc.ErrorDetails rather than free-form strings:

st := status.New(codes.InvalidArgument, "invalid create request")
st, _ = st.WithDetails(&errdetails.BadRequest{
	FieldViolations: []*errdetails.BadRequest_FieldViolation{
		{Field: "items", Description: "must contain at least one item"},
	},
})
return nil, st.Err()

Logging interceptors are a classic data-leak path. Never log whole request messages (they contain PII), and redact metadata by allowlist:

var loggableMD = map[string]bool{"x-request-id": true, "user-agent": true, ":authority": true}
// authorization, cookie, x-api-key and friends never make it into the log line.

7.8 Graceful shutdown

go func() {
	<-ctx.Done()
	hs.SetServingStatus("", healthpb.HealthCheckResponse_NOT_SERVING) // fail readiness first
	time.Sleep(cfg.DrainDelay)                                        // let the LB notice
	stopped := make(chan struct{})
	go func() { srv.GracefulStop(); close(stopped) }()
	select {
	case <-stopped:
	case <-time.After(20 * time.Second):
		srv.Stop() // hard stop: in-flight streams die rather than block the deploy
	}
}()

On Kubernetes, use the native gRPC probe rather than a sidecar binary:

readinessProbe:
  grpc:
    port: 8443
    service: acme.orders.v1.OrderService
  periodSeconds: 5

Note that the kubelet's gRPC probe does not present a client certificate, so with RequireAndVerifyClientCert it will fail. The usual solutions are a second plaintext listener bound to localhost for probes, or an exec probe using grpc_health_probe with the pod's own certs.

8. Client-side robustness

Service config is JSON attached to the channel; it configures load balancing, per-method timeouts and retries without touching call sites.

const serviceConfig = `{
  "loadBalancingConfig": [{"round_robin":{}}],
  "methodConfig": [
    {
      "name": [{"service": "acme.orders.v1.OrderService", "method": "GetOrder"}],
      "timeout": "2s",
      "retryPolicy": {
        "maxAttempts": 4,
        "initialBackoff": "0.1s",
        "maxBackoff": "1s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": ["UNAVAILABLE", "RESOURCE_EXHAUSTED"]
      }
    },
    {
      "name": [{"service": "acme.orders.v1.OrderService", "method": "CreateOrder"}],
      "timeout": "5s"
    }
  ]
}`

conn, err := grpc.NewClient(target,
	grpc.WithTransportCredentials(creds),
	grpc.WithDefaultServiceConfig(serviceConfig),
	grpc.WithKeepaliveParams(keepalive.ClientParameters{
		Time:                1 * time.Minute, // must be >= the server's MinTime, or you get GOAWAY
		Timeout:             20 * time.Second,
		PermitWithoutStream: false,
	}),
)

The failure modes, in rough order of how often they show up:

  • Retrying non-idempotent RPCs. CreateOrder above has no retry policy on purpose. Give it an idempotency_key and dedupe server-side before you consider retrying it.
  • Retry storms. maxAttempts is capped at 5 by default and the whole channel has a retry throttle; keep it. A client that retries four times into an overloaded server turns a brownout into an outage.
  • Missing dns:///. Without the scheme, the default resolver may resolve once at startup and never see new pods. With it, grpc-go re-resolves on transient failure.
  • Client keepalive Time below the server's MinTime. The server sends GOAWAY with ENHANCE_YOUR_CALM and the connection dies for reasons that look like a network fault.
  • UNAVAILABLE treated as fatal. It is the normal code for "connection not ready yet"; with a lazy NewClient the first call of the process can legitimately see it.

9. The same thing in other languages

The APIs differ; the model does not. In every language you are doing four things: load identity material, pin the trust root, require the peer certificate, attach per-call credentials.

9.1 Python

import grpc
from concurrent import futures

def _read(p): return open(p, "rb").read()

# ---- server ----
creds = grpc.ssl_server_credentials(
    private_key_certificate_chain_pairs=[(_read("server.key"), _read("server.crt"))],
    root_certificates=_read("ca.crt"),
    require_client_auth=True,          # <- mutual TLS
)
server = grpc.server(
    futures.ThreadPoolExecutor(max_workers=16),
    options=[
        ("grpc.max_receive_message_length", 4 * 1024 * 1024),
        ("grpc.max_concurrent_streams", 256),
        ("grpc.keepalive_time_ms", 120000),
    ],
    interceptors=[AuthInterceptor()],
)
orders_pb2_grpc.add_OrderServiceServicer_to_server(OrderService(), server)
server.add_secure_port("0.0.0.0:8443", creds)   # NEVER add_insecure_port in production
server.start()

# ---- client ----
channel_creds = grpc.ssl_channel_credentials(
    root_certificates=_read("ca.crt"),
    private_key=_read("client.key"),
    certificate_chain=_read("client.crt"),
)
call_creds = grpc.access_token_call_credentials(token)   # refuses to run on insecure channels
combined = grpc.composite_channel_credentials(channel_creds, call_creds)

with grpc.secure_channel("orders.acme.svc:8443", combined) as channel:
    stub = orders_pb2_grpc.OrderServiceStub(channel)
    order = stub.GetOrder(orders_pb2.GetOrderRequest(id=order_id), timeout=2.0)

Server-side identity in Python comes from context.auth_context(), whose x509_common_name / x509_subject_alternative_name keys are populated only when the peer certificate was verified.

9.2 Java

// server
ServerCredentials creds = TlsServerCredentials.newBuilder()
    .keyManager(new File("server.crt"), new File("server.key"))
    .trustManager(new File("ca.crt"))
    .clientAuth(TlsServerCredentials.ClientAuth.REQUIRE)   // mutual TLS
    .build();

Server server = Grpc.newServerBuilderForPort(8443, creds)
    .addService(new OrderServiceImpl())
    .addService(ProtoReflectionService.newInstance())   // dev only
    .intercept(new AuthInterceptor())
    .maxInboundMessageSize(4 * 1024 * 1024)
    .build()
    .start();

// client
ChannelCredentials clientCreds = TlsChannelCredentials.newBuilder()
    .keyManager(new File("client.crt"), new File("client.key"))
    .trustManager(new File("ca.crt"))
    .build();

ManagedChannel channel = Grpc.newChannelBuilder("orders.acme.svc:8443", clientCreds)
    .defaultServiceConfig(serviceConfigMap)
    .enableRetry()
    .build();

OrderServiceGrpc.OrderServiceBlockingStub stub = OrderServiceGrpc.newBlockingStub(channel)
    .withDeadlineAfter(2, TimeUnit.SECONDS)
    .withCallCredentials(new BearerToken(tokenSupplier));

9.3 Node.js

const grpc = require('@grpc/grpc-js');

// server
const creds = grpc.ServerCredentials.createSsl(
  fs.readFileSync('ca.crt'),
  [{ private_key: fs.readFileSync('server.key'), cert_chain: fs.readFileSync('server.crt') }],
  true,                       // checkClientCertificate -> mutual TLS
);
server.bindAsync('0.0.0.0:8443', creds, cb);

// client
const channelCreds = grpc.credentials.createSsl(
  fs.readFileSync('ca.crt'), fs.readFileSync('client.key'), fs.readFileSync('client.crt'));
const callCreds = grpc.credentials.createFromMetadataGenerator((_params, cb) => {
  const md = new grpc.Metadata();
  md.set('authorization', `Bearer ${token()}`);
  cb(null, md);
});
const client = new OrderService(
  'orders.acme.svc:8443',
  grpc.credentials.combineChannelCredentials(channelCreds, callCreds),
  { 'grpc.max_receive_message_length': 4 * 1024 * 1024 },
);

10. Testing and debugging

10.1 In-process tests with bufconn

Fast, hermetic, no ports, no certs -- the one place plaintext is correct:

func TestGetOrder(t *testing.T) {
	lis := bufconn.Listen(1024 * 1024)
	srv := grpc.NewServer(grpc.ChainUnaryInterceptor(testAuthInterceptor))
	ordersv1.RegisterOrderServiceServer(srv, &Server{store: fakeStore()})
	go func() { _ = srv.Serve(lis) }()
	t.Cleanup(srv.Stop)

	conn, err := grpc.NewClient("passthrough:///bufnet",
		grpc.WithContextDialer(func(ctx context.Context, _ string) (net.Conn, error) {
			return lis.DialContext(ctx)
		}),
		grpc.WithTransportCredentials(insecure.NewCredentials()),
	)
	require.NoError(t, err)
	t.Cleanup(func() { _ = conn.Close() })

	_, err = ordersv1.NewOrderServiceClient(conn).GetOrder(ctx,
		&ordersv1.GetOrderRequest{Id: "missing"})
	require.Equal(t, codes.NotFound, status.Code(err))
}

The passthrough:/// scheme is required with grpc.NewClient, which otherwise sends bufnet to the DNS resolver.

Then add the tests that actually prove your security config, against a real listener with real certs:

func TestRejectsCallerWithoutClientCert(t *testing.T) { /* expect handshake failure */ }
func TestRejectsCallerFromUntrustedCA(t *testing.T)   { /* expect unknown authority */ }
func TestRejectsUnauthorizedMethod(t *testing.T)      { /* expect codes.PermissionDenied */ }
func TestRejectsExpiredToken(t *testing.T)            { /* expect codes.Unauthenticated */ }

10.2 Poking a live service

# Reflection on (dev): discover everything
grpcurl -cacert ca.crt -cert client.crt -key client.key \
  orders.acme.svc:8443 list
grpcurl -cacert ca.crt -cert client.crt -key client.key \
  orders.acme.svc:8443 describe acme.orders.v1.OrderService.CreateOrder

# Reflection off (prod): supply the schema yourself
buf build -o api.binpb --as-file-descriptor-set
grpcurl -protoset api.binpb -cacert ca.crt -cert client.crt -key client.key \
  -H 'authorization: Bearer '"$TOKEN" \
  -d '{"id": "7f1c2d0e-1d2b-4a3e-9f10-6c2a5b8e4d11"}' \
  orders.acme.svc:8443 acme.orders.v1.OrderService/GetOrder

# Streaming input: newline-delimited JSON on stdin
echo '{"sku":"A"}
{"sku":"B"}' | grpcurl -protoset api.binpb -d @ ... OrderService/UploadReceipts

# Browser UI over the same connection
grpcui -cacert ca.crt -cert client.crt -key client.key orders.acme.svc:8443

# Load test, including the handshake cost
ghz --insecure=false --cacert ca.crt --cert client.crt --key client.key \
    --proto proto/acme/orders/v1/order.proto \
    --call acme.orders.v1.OrderService.GetOrder \
    -d '{"id":"7f1c2d0e-1d2b-4a3e-9f10-6c2a5b8e4d11"}' \
    -c 50 -n 20000 orders.acme.svc:8443

10.3 When it does not connect

# grpc-go's own trace: handshakes, resolver, subchannel state
GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=99 ./orders-server

# Is TLS even right? Check ALPN is h2 and the CA list is what you think.
openssl s_client -connect orders.acme.svc:8443 -alpn h2 -showcerts </dev/null

# Decode a cert you were handed
openssl x509 -in client.crt -noout -subject -issuer -dates -ext subjectAltName

A decoder ring for the errors you will actually see:

  • transport: authentication handshake failed: x509: certificate signed by unknown authority -- the client's RootCAs does not contain the CA that signed the server cert.
  • x509: certificate is valid for X, not Y -- ServerName/target host is not in the server certificate's SANs.
  • tls: client didn't provide a certificate -- server is on RequireAndVerifyClientCert and the client has no Certificates. This is mTLS working.
  • remote error: tls: bad certificate -- the server rejected your client cert: wrong CA, expired, or missing the clientAuth EKU.
  • Unimplemented -- package/service name mismatch between the client's stub and the server's registration, or a proxy stripped the path.
  • ENHANCE_YOUR_CALM ... too_many_pings -- client keepalive is more aggressive than the server's MinTime.
  • ResourceExhausted: grpc: received message larger than max -- raise the limit deliberately or switch to streaming; do not set it to math.MaxInt32.

channelz is the last resort for connection-state mysteries -- register it with channelz.RegisterChannelzServiceToServer(srv) and inspect with grpcdebug.

11. Observability

Use the OpenTelemetry stats handlers, not the old interceptors (which are deprecated in otelgrpc): they see connection-level events that interceptors cannot.

import "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"

grpc.NewServer(grpc.StatsHandler(otelgrpc.NewServerHandler()))
grpc.NewClient(target, grpc.WithStatsHandler(otelgrpc.NewClientHandler()))

Trace context propagates in metadata automatically, so a traceparent set at the edge follows the call across every hop. What to watch:

  • Rate of each grpc_code per method -- a rising Unauthenticated rate usually means a rotation broke, and a rising PermissionDenied rate means either a misconfigured caller or someone probing.
  • DeadlineExceeded on the client vs Internal on the server: if only the client sees errors, your timeouts are too tight, not your server too slow.
  • Certificate expiry as a gauge. Expiry is the single most common self-inflicted gRPC outage; alert at 30% of remaining lifetime, not at 24 hours.

12. Pre-deploy checklist

  1. Plaintext is impossible: no insecure.NewCredentials(), no add_insecure_port, no InsecureSkipVerify outside tests, and CI greps for all three.
  2. ClientAuth: tls.RequireAndVerifyClientCert (or the language equivalent) on every server, with ClientCAs pinned to your internal CA.
  3. RootCAs pinned on every client -- never the system trust store for internal traffic.
  4. MinVersion: tls.VersionTLS13 where the fleet allows it, TLS 1.2 as the floor otherwise.
  5. Certificates rotate without a restart, and a negative test proves an unauthenticated caller is rejected.
  6. Caller identity is read from VerifiedChains, never PeerCertificates.
  7. Authorization is deny-by-default at the method level, plus object-level checks in handlers, with NotFound rather than PermissionDenied where existence is sensitive.
  8. PerRPCCredentials.RequireTransportSecurity() returns true.
  9. MaxRecvMsgSize, MaxSendMsgSize, MaxConcurrentStreams and MaxHeaderListSize are all set explicitly.
  10. Keepalive enforcement policy set; MaxConnectionAge bounds connection lifetime.
  11. Every client sets a deadline; the server enforces a ceiling; handlers derive downstream contexts from the incoming one.
  12. Panic recovery returns codes.Internal with no detail, and no handler returns a wrapped internal error to the caller.
  13. Request payloads and authorization metadata are never logged.
  14. Reflection is disabled (or authenticated) in production.
  15. buf lint and buf breaking run in CI, and generated code is verified up to date.
  16. Health service registered and wired to a readiness probe that works under mTLS.
  17. Certificate expiry is monitored and alerted on.

Sections 5 and 6 are the ones to get right; the rest is ordinary service engineering that you already know how to do.

If you keep one thing from this, keep the four-layer split: the transport proves which workload is calling, the token proves on whose behalf, the policy decides which methods it may reach, and the handler decides which rows it may see. Each layer answers a question the one below it cannot. Drop any single layer and the service still works perfectly, right up until someone goes looking for the gap.

Comments (0)

Be the first to comment.