Building Resilient Distributed Systems with go-zero

If you’re a Go developer wrestling with the complexities of building scalable, reliable microservices, go-zero is a framework you need to know. With over 32,781 stars on GitHub, go-zero is not just another web framework—it’s a comprehensive toolkit designed from the ground up for high-concurrency, distributed systems. Created by Tal-tech, it combines a powerful code generator (goctl), a robust RPC and HTTP stack, and a suite of production-ready components like service discovery, circuit breaking, and rate limiting. Its philosophy is to solve the hard problems of distributed systems out of the box, allowing you to focus on business logic rather than boilerplate infrastructure code. Whether you’re building a high-traffic e-commerce platform, a real-time messaging system, or a complex backend for a fintech application, go-zero provides the patterns and tools to ensure stability and performance under load. In this post, we’ll dive deep into its architecture, walk through practical setup, and build real-world examples that showcase its power.

Key Features

go-zero’s strength lies in its batteries-included, yet modular design. Here are the major features that set it apart:

goctl Code Generator: This is go-zero’s killer feature. Instead of manually writing repetitive server, client, and DTO (Data Transfer Object) code, goctl scaffolds complete, production-ready services from simple API or proto definitions. It generates directory structures, handler logic, models, configuration files, and even Dockerfiles. This enforces consistency and drastically reduces development time.
Unified RPC & HTTP Stack: go-zero treats HTTP REST APIs and gRPC services as first-class citizens with a nearly identical programming model. You define your service logic once (in a logic layer), and goctl can generate both an HTTP server (with routing, middleware, validation) and a gRPC server (with streaming support) that call into the same logic. This promotes a clean architecture and easy protocol evolution.
Built-in Service Discovery & Registration: go-zero includes an integrated etcd-based service registry (zookeeper is also supported). Your services automatically register themselves on startup and discover dependencies via a simple, consistent naming scheme (etcd://service-name). This eliminates the need for external service mesh sidecars for basic service-to-service communication.
Resilience Patterns (Circuit Breaker, Rate Limiting, Timeout): Every RPC and HTTP call is wrapped with production-grade resilience. rest.ResilientClient and rpc.ResilientClient provide automatic retries with backoff, circuit breaking (using gobreaker), rate limiting (token bucket), and timeouts—all configurable via YAML. This is crucial for preventing cascading failures in a distributed system.
Observability & Tracing: go-zero has deep integration with OpenTelemetry. It automatically propagates trace contexts across HTTP and gRPC calls, generates spans for each handler, and provides built-in metrics (latency, QPS, errors) via Prometheus. The ztrace package makes distributed tracing effortless.
High-Performance Load Balancing: For outbound RPC calls, go-zero implements a sophisticated load balancer that supports round-robin, random, and consistent hashing. It performs real-time health checks on registered endpoints and automatically evicts unhealthy instances, ensuring traffic only goes to available nodes.
Configuration Management: A centralized, type-safe configuration system (conf.Load) loads from multiple sources (YAML files, environment variables, command-line flags) with priority. It supports hot-reloading and environment-specific configs (config.yaml, config.dev.yaml), which is essential for 12-factor app compliance.
Caching & Synchronization: Includes a multi-level cache (lruc, redis) with automatic cache warming and a distributed lock (redsync) based on Redis. These are abstracted behind simple interfaces, making it easy to add caching or leader election to your services.

Comparison with Standard Go: The standard library (net/http, google.golang.org/grpc) gives you raw building blocks. You’d need to manually integrate libraries like go-kit, uber-go/zap, go-redis, and write extensive boilerplate for service discovery, circuit breaking, and config management. go-zero packages all this into a cohesive, convention-over-configuration framework, drastically reducing operational complexity and ensuring best practices are applied by default.

Installation and Setup

go-zero requires Go 1.16 or later. The primary installation is for the code generator and core libraries.

1# Install the latest go-zero and goctl
2go install -u github.com/tal-tech/go-zero/tools/goctl@latest
3
4# Verify installation
5goctl --version
6# Expected output: goctl version v1.6.0 (or similar)

Setup is convention-driven. After installation, you use goctl to create new projects. There’s no complex global configuration. The framework relies on a standard directory layout:

etc/: Configuration files (YAML)
internal/: Application code (logic, models, handlers)
api/: REST API definition files (.api)
rpc/: gRPC service definition files (.proto)

To start a new REST API service:

1goctl api new user-service

This creates a complete project structure. You can verify by navigating into the directory and running:

1cd user-service
2go mod tidy
3go build -o user-service .
4./user-service -f etc/user-api.yaml

The service will start on the port defined in etc/user-api.yaml (default 8888). You can test it with curl http://localhost:8888/hello.

Basic Usage

Let’s create a minimal “Hello World” REST API. First, generate a new service:

1goctl api new hello-world
2cd hello-world

The generator creates an api/hello.api file. Replace its content with:

 1syntax = "v1"
 2
 3info (
 4    title: "Hello World API"
 5    desc: "A minimal go-zero API"
 6    author: "dev"
 7    version: "v1.0"
 8)
 9
10type (
11    HelloRequest {
12        Name string `json:"name"`
13    }
14
15    HelloResponse {
16        Message string `json:"message"`
17    }
18)
19
20@server (
21    prefix: /hello
22    group: default
23)
24service hello-api {
25    @doc "Say hello"
26    @handler hello
27    get /say (HelloRequest) returns (HelloResponse)
28}

Now, generate the code:

1goctl api go -api api/hello.api -dir .

This generates the server, handler, and logic. The core logic lives in internal/logic/hellologic.go. Modify internal/logic/hellologic.go:

 1package logic
 2
 3import (
 4    "context"
 5    "fmt"
 6
 7    "hello-world/internal/svc"
 8    "hello-world/internal/types"
 9
10    "github.com/tal-tech/go-zero/core/logx"
11)
12
13type HelloLogic struct {
14    logx.Logger
15    ctx    context.Context
16    svcCtx *svc.ServiceContext
17}
18
19func NewHelloLogic(ctx context.Context, svcCtx *svc.ServiceContext) *HelloLogic {
20    return &HelloLogic{
21        Logger: logx.WithContext(ctx),
22        ctx:    ctx,
23        svcCtx: svcCtx,
24    }
25}
26
27func (l *HelloLogic) Say(req *types.HelloRequest) (*types.HelloResponse, error) {
28    // Simple business logic: create a greeting
29    message := fmt.Sprintf("Hello, %s! Welcome to go-zero.", req.Name)
30    l.Infof("Generated greeting for: %s", req.Name)
31    return &types.HelloResponse{
32        Message: message,
33    }, nil
34}

The svc.ServiceContext is defined in internal/config/config.go and is where you would inject database connections, caches, etc. For this example, it’s empty.

Finally, run the server:

1go run hello.go -f etc/hello-api.yaml

Expected Output & Test:
The server starts, logging something like:

12023/10/27 10:00:00.000 [INFO]  starting server at 0.0.0.0:8888

In another terminal, test the endpoint:

1curl "http://localhost:8888/hello/say?name=World"

You receive:

1{"message":"Hello, World! Welcome to go-zero."}

Explanation:

API Definition (.api file): This declarative file defines your service’s contract. goctl uses it to generate all boilerplate.
Generated Structure: goctl creates a clean layered architecture:
- handler: HTTP-specific code (unmarshaling request, sending response).
- logic: Pure business logic, independent of transport (HTTP/gRPC). It receives a typed ServiceContext.
- model: Data models and database access (if defined).
- svc: ServiceContext struct for dependency injection.
NewHelloLogic: The constructor receives a context.Context and the shared svcCtx. This is where you’d get your DB or Redis client from svcCtx.
Say method: Contains the core logic. It’s a simple function that takes a request DTO and returns a response DTO or an error. Returning an error automatically translates to an appropriate HTTP 500 (or gRPC status) response.
Configuration: The -f etc/hello-api.yaml flag loads the server configuration (port, timeout, etc.). You can change the port there.

Real-World Examples

Example 1: Full Microservices System (User & Order Services with gRPC & HTTP)

This example demonstrates a canonical go-zero microservice pattern: an HTTP API gateway that calls a gRPC backend service. We’ll create a user service with both HTTP and gRPC endpoints, and an order service that calls the user service via RPC, showcasing service discovery and resilience.

Project Structure:

 1microservices-demo/
 2├── user/
 3│   ├── api/
 4│   │   └── user.api
 5│   ├── rpc/
 6│   │   ├── user.proto
 7│   │   └── internal/
 8│   │       └── logic/
 9│   ├── etc/
10│   │   ├── user-api.yaml
11│   │   └── user-rpc.yaml
12│   └── go.mod
13├── order/
14│   ├── api/
15│   │   └── order.api
16│   ├── rpc/
17│   │   ├── order.proto
18│   │   └── internal/
19│   │       └── logic/
20│   ├── etc/
21│   │   ├── order-api.yaml
22│   │   └── order-rpc.yaml
23│   └── go.mod
24└── docker-compose.yaml (for etcd)

Step 1: User Service (Both HTTP & RPC)

Define RPC (user/rpc/user.proto):

 1syntax = "proto3";
 2
 3package user;
 4
 5option go_package = "github.com/your-username/microservices-demo/user/rpc/internal/pb";
 6
 7message User {
 8    int64 id = 1;
 9    string name = 2;
10    string email = 3;
11}
12
13message GetUserReq {
14    int64 id = 1;
15}
16
17message GetUserResp {
18    User user = 1;
19}
20
21service UserService {
22    rpc GetUser (GetUserReq) returns (GetUserResp);
23}

Generate RPC Code:

1cd user/rpc
2goctl rpc protoc user.proto --go_out=. --go-grpc_out=. --zrpc_out=.

This generates internal/pb/user.pb.go and the server skeleton.

Implement RPC Logic (user/rpc/internal/logic/getuserlogic.go):

 1package logic
 2
 3import (
 4    "context"
 5    "fmt"
 6
 7    "github.com/tal-tech/go-zero/core/logx"
 8    "github.com/your-username/microservices-demo/user/rpc/internal/svc"
 9    "github.com/your-username/microservices-demo/user/rpc/internal/pb"
10)
11
12type GetUserLogic struct {
13    logx.Logger
14    ctx    context.Context
15    svcCtx *svc.ServiceContext
16}
17
18func NewGetUserLogic(ctx context.Context, svcCtx *svc.ServiceContext) *GetUserLogic {
19    return &GetUserLogic{
20        Logger: logx.WithContext(ctx),
21        ctx:    ctx,
22        svcCtx: svcCtx,
23    }
24}
25
26// In a real app, this would query a database.
27func (l *GetUserLogic) GetUser(in *pb.GetUserReq) (*pb.GetUserResp, error) {
28    l.Infof("Fetching user with ID: %d", in.Id)
29    // Simulated DB fetch
30    user := &pb.User{
31        Id:    in.Id,
32        Name:  fmt.Sprintf("User %d", in.Id),
33        Email: fmt.Sprintf("user%[email protected]", in.Id),
34    }
35    return &pb.GetUserResp{User: user}, nil
36}

Define HTTP API (user/api/user.api):

 1type (
 2    UserReq {
 3        Id int64 `path:"id"`
 4    }
 5
 6    UserResp {
 7        Id    int64  `json:"id"`
 8        Name  string `json:"name"`
 9        Email string `json:"email"`
10    }
11)
12
13@server (
14    prefix: /api/v1
15    group: user
16)
17service user-api {
18    @doc "Get user by ID via HTTP (calls RPC internally)"
19    @handler GetUser
20    get /user/:id (UserReq) returns (UserResp)
21}

Generate HTTP code: goctl api go -api api/user.api -dir .

Wire HTTP to RPC in Logic (user/internal/logic/getuserlogic.go for HTTP):

 1package logic
 2
 3import (
 4    "context"
 5    "fmt"
 6
 7    "github.com/tal-tech/go-zero/core/logx"
 8    "github.com/tal-tech/go-zero/rest/httpx"
 9    "github.com/your-username/microservices-demo/user/rpc/internal/pb"
10    "github.com/your-username/microservices-demo/user/rpc/internal/svc"
11    "github.com/your-username/microservices-demo/user/internal/types"
12)
13
14// This is the HTTP handler's logic. It calls the RPC client.
15func (l *UserLogic) GetUser(req *types.UserReq) (*types.UserResp, error) {
16    l.Infof("HTTP request for user ID: %d", req.Id)
17
18    // Call the RPC service using the resilient client from svcCtx
19    rpcResp, err := l.svcCtx.UserRpcClient.GetUser(context.Background(), &pb.GetUserReq{
20        Id: req.Id,
21    })
22    if err != nil {
23        l.Errorf("RPC call failed: %v", err)
24        return nil, err
25    }
26
27    // Map RPC response to HTTP response type
28    resp := &types.UserResp{
29        Id:    rpcResp.User.Id,
30        Name:  rpcResp.User.Name,
31        Email: rpcResp.User.Email,
32    }
33    return resp, nil
34}