Go Project Layout

This is my understanding of how to structure Go project layout. The most important part is how to organize your Go code into packages.

Motivation

A good project layout will make your source code easy to understand, easy to test, and easy to maintain.
Kat Zien has a very good summary(Slides) for this, so I'd like to refer it directly as below:

Questions, decisions

• Should I put everything in the main package?
• Should I start with one package and extract other packages over time?
• How do I decide if something should be in its own package?
• Should I just use a framework?
• What's the programming paradigm for Go?
• Microservices or monolith?
• How much should be shared between packages?

Why should we care?

Because if Go is going to be a language that companies invest in for the long term, the maintenance of Go programs, the ease of which they can change, will be a key factor in their decision. - Dave Cheney, Golang UK 2016 keynote

Good structure goals

• Consistent.
• Easy to understand, navigate and reason about. ("makes sense") Easy to change, loosely-coupled.
• Easy to test.
• "As simple as possible, but no simpler."
• Design reflects exactly how the software works.
• Structure reflects the design exactly.

Demo

# shell 1
$ go version
go version go1.13.1 linux/amd64
$
$ cd $GOPATH
$ 
$ # app 1: reverse echo
$ go get -u github.com/wangyoucao577/go-project-layout/cmd/echor
$ ./bin/echor "hello world"
dlrow olleh
$
$ # app 2: a simple diagnosis service for server diagnosis. 
$ #    return `Hostname/IP/CPUs/RemoteAddr...` by `HTTP`. 
$ #    sample request: `http://localhost:8000/diagnosis?diagnosis=ping`
$ go get -u github.com/wangyoucao577/go-project-layout/cmd/diagnosis
$ ./bin/diagnosis -alsologtostderr
I1007 18:50:05.550952    3769 main.go:33] Listen on :8000

# shell 2
$ curl "http://localhost:8000/diagnosis?diagnosis=ping"
{
  "Hostname":"server",
  "IP Addresses":[
    "192.168.29.201/24",
    "fe80::1c20:479:9094:4327/64",
    "192.168.141.1/24",
    "fe80::8002:2d87:c4f3:4aab/64",
    "192.168.128.1/24",
    "fe80::eca9:cfa0:9443:b8ff/64",
    "192.168.44.209/28",
    "fe80::8551:306e:7e7:6faf/64"
  ],
  "CPUs":8,
  "Remote Endpoint":"127.0.0.1:64717"
}

Note

This topic is kind of best practices. I'd like to discuss some of them and my opinion, then maybe deep into more best practices in the future.

Top dir /cmd

[My opinion: Strongly recommended]
The cmd layout pattern is very useful when you need to have more than one application binary.

• Each binary gets a subdirectory (e.g., your_project/cmd/your_app).
  • The subdirectory is the main package for your application.
  • Don't put a lot of code in the main package. Instead, it should only used to initialises and ties everything together.
• This patterns also helps you keep your project/package go gettable.
  • It means you can use the go get command to fetch (and install) your project, its applications and its libraries (e.g., go get github.com/your_github_username/your_project/cmd/your_app).
• The official Go tool is one example of the cmd layout patter. A number of other well known projects use the same pattern: Kubernetes, Docker, Prometheus, Influxdb.

Top dir /pkg

[My opinion: NOT good enough, use it if good for you]

• People also recommend to set up a /pkg top dir in Go project to put your public libraries.
  • These libraries can be used internally by your application.
  • They can also be used by external projects. It's an informal contract between you and other external users of your code. Other projects will import these libraries expecting them to work.
• But I think it's NOT a good enough idea because
  • It brings confusion since Go workspaces have a directory with the same name and that directory has a different purpose(it’s used to store object files for the packages the Go compiler builds).
  • The meaning of "public libraries" is not clear enough.
    • Even many well known projects use this pattern (Kubernetes, Docker, Grafana, Influxdb, Etcd), Docker and Etcd comment the meaning of "public libraries" again. Both of them limit it only as utility packages, and these utility packages possible to be moved out into its own repository in the future:
      • Docker: pkg/ is a collection of utility packages used by the moby project without being specific to its internals.
      • Etcd: pkg/ is a collection of utility packages used by etcd without being specific to etcd itself. A package belongs here only if it could possibly be moved out into its own repository in the future.

Group packages by dependency

[My opinion: Strongly recommended]

• Domain types
  • Domain Types are types that model your business functionality and objects.
  • Ben Johnson - Standard Package Layout recommend to place your into root package. This package only contains simple data types, so it will be extremely simple.
  • In my opinion, either place them into root package or place them into a individual subpackage are ok. The most important part here is this package should be extermely simple, and not depend on any other package in application!
• Each package should only has a single purpose.
• Packages names describe their purpose.
• When necessary, use a descriptive parent package and several children implementing the functionality.
  • like the encoding package.
• main package ties together dependencies

More best practices

• Follow conventions
  • Be similar to the standard library and other popular packages
  • Don't surprise people
  • Be obvious, not clever
• Use Go modules
• Avoid global scope and init()

References

• GopherCon UK 2018: Kat Zien - How do you structure your Go apps?
  • This's an amazing talk for the Go project layout I think.
  • Slides
  • Code: katzien/go-structure-examples
• Standard Go Project Layout
  • No, it's NOT a golang standard!!! See more in Delete this repo #36 and This repository has multiple serious issues #41.
  • It's just an organization name "golang-standards" used by some one to manage these repos. It's harmful if you really think it's a standard and try to always follow it.
  • On the other hand, I think this repo still have some benifits: collects many resources for the "golang-project-layout" question, gives suggestion for how to do it, etc.
  • Try to read it as a post, think about what's valueable for you.
  • More resources metioned in #36
• Ben Johnson - Standard Package Layout
• Ben Johnson - Building WTF Dial
• Kyle C. Quest - Go Project Layout
• GopherCon Russia 2018: Ashley McNamara + Brian Ketelsen - Go best practices
  • slides
• Golang UK Conference 2017 | Mat Ryer - Writing Beautiful Packages in Go
• CodeBear801 - Golang Package layout
• golang/go
• golang/tools
• moby/moby
• kubernetes/kubernetes
• etcd-io/etcd
• influxdata/influxdb
• prometheus/prometheus