The most popular Go dependency is…

2026-01-05T00:00:00+00:00

…

As you know, destination is not as important as the journey, so now that we got this out of the way, bear with me for the rest of this article, and I’ll give you the top 10, and many more stats 😊 You might even learn a few things along the way!

Without usage statistics, finding useful and reliable dependencies can be a bit of a challenge. In the Go community, we basically have to rely either:

on "brand" name reputation: there are some very well known packages (gin</a>, cobra</a>, testify</a>…) and organizations (Google</a>, gorilla</a>…) that we can trust,</li>
or on side metrics such as number of GitHub stars, number of open issues, last activity, and more.</li> </ul>
All of this can sometimes help get a feeling of how widely used and trusted a Go module is, but we could get more. What I personally want is knowing how many times a module is actually required as a project dependency to get a feeling of how "battle-tested" a library is. But to do this, I would need to build a graph of the whole (open source) ecosystem, which would be insane… You can see where this is going 😉
The source code for this project is available on github.com/Thiht/go-stats</a>. It uses Go as a primary language (obviously!), and Neo4j</a> as a database. </aside>
Mapping the Go ecosystem</h2>
My first idea was to build a list of repositories (a seed) to use as a starting point. The goal was to read the dependencies of these modules from their go.mod</code>, download each of them, read the dependencies of these modules from their go.mod</code>, download each of them, read the dependencies of these modules from their go.mod</code>, download each of… well, you know the deal.
I implemented this idea on the v1</code></a> branch of my repository using mainly Github-Ranking</a> and awesome-go</a> as sources for building the seed. I ultimately abandoned the idea because of a few shortcomings:
the sample is largely incomplete,</li> cloning so many Git repositories to find their go.mod</code> is reaaally painful and slow,</li> and it’s particularly biased towards repositories hosted on GitHub.</li> </ul> Luckily for me, I came up with a second idea: the Go modules ecosystem relies on a centralized public proxy, so surely they expose some information on these modules. And they in fact do so! The proxy APIs are documented on proxy.golang.org</a>: proxy.golang.org</a> exposes metadata on each module (versions, latest, mod file…),</li> index.golang.org</a> exposes a feed of all the published module versions since the introduction of the Go proxy (2019-04-10T19:08:52.997264Z</code>, if you want to make sure not to forget its birthday).</li> </ul> I used this information to locally download the whole index (module names and versions) since 2019. The downloaded data is available in goproxy-modules</a>. This can be used as a local immutable cache. For the full implementation details, see: list-goproxy-modules.go</code></a></li> goproxy.go</code></a></li> </ul> </aside> With all this data available locally, the seed is now pretty much exhaustive, and more suitable for data analysis. The processing now simply consists of iterating over every single module, downloading their go.mod</code> file and listing their dependencies. The resulting graph can then trivially be inserted in a specialized graph database like Neo4j. Deep diving</h2> Neo4j is a graph oriented database. It means that unlike relational databases, it works on... graphs. The primary way to store data in Neo4j is using nodes and relationships. This specialized data structure makes it extremely simple to model, and more importantly query huge graphs. If you want to experiment with Neo4j, I would recommend using the go-stats docker-compose.yml</code></a> file. You can then open localhost:7474</a> (no credentials needed) to use the Neo4j browser. </aside> Neo4j, like many NoSQL databases, is schemaless, meaning you don't need to define a schema before creating data. That doesn't mean we don't need a schema, so let's see what we need! erDiagram Module { string name string version } Module ||--o{ Module : DEPENDS_ON </pre> A Go module is basically identified by its name (eg. github.com/stretchr/testify</code> or go.yaml.in/yaml/v4</code>) and its version. Each module can depend on other modules. We can add more properties to our nodes later on as needed. Creating nodes</h3> Neo4j uses Cypher</a> as a query language. Inserting data with Cypher can be done with the CREATE</code></a> clause, but in go-stats I've decided to use the MERGE</code></a> clause instead because it behaves as an upsert, letting you update or do nothing in case a node already exists. The basic Cypher query to upsert a module node is: MERGE (m:Module { name: $name, version: $version }) RETURN m</code></pre> :Module</code> is a label attached to the node. You can think of it as the type of the node. name</code> and version</code> are properties of the node, they're the data belonging to each specific node. To make sure we can't create multiple nodes with the same name-version pair, a unicity constraint is needed: CREATE CONSTRAINT module_identity IF NOT EXISTS FOR (m:Module) REQUIRE (m.name, m.version) IS UNIQUE</code></pre> Thanks to this constraint, if MERGE</code> is called a second time with the same name</code> and version</code> properties, it won't do anything. Creating relationships</h3> We can then create the dependency relationships between our module nodes: MATCH (dependency:Module { name: $dependencyName, version: $dependencyVersion }) MATCH (dependent:Module { name: $dependentName, version: $dependentVersion }) MERGE (dependent)-[:DEPENDS_ON]->(dependency) RETURN dependency, dependent</code></pre> This query will: find modules that were created earlier using the MATCH</code></a> clause and assign them to dependency</code> and dependent</code>,</li> create the directed relationship between them using the -[:DEPENDS_ON]-></code> syntax.</li> </ol> The Go index is naturally sorted chronologically. So as long as we iterate over it sequentially, it means that if a module (dependent) depends on another module (dependency), then dependency was necessarily added to the graph before dependent. If this condition doesn't hold for some reason (if we were to decide to parallelize the insertions for example), we could simply rewrite the query as: MERGE (dependency:Module { name: $dependencyName, version: $dependencyVersion }) MERGE (dependent:Module { name: $dependentName, version: $dependentVersion }) MERGE (dependent)-[:DEPENDS_ON]->(dependency) RETURN dependency, dependent</code></pre> Using MERGE</code> instead of MATCH</code> would ensure the node gets created if it doesn't exist already. These queries are simplified (but close!) variants of what I actually did in go-stats. The main difference is that I enriched the nodes with some additional properties: version timestamp,</li> latest version,</li> semantic version splitting (major, minor, patch, label),</li> host, organisation, and more.</li> </ul> For the full implementation details, see: process-modules.go</code></a> </aside> Digging into the graph</h2> After running go-stats for a few days, I ended up with a graph of roughly 40 million nodes, and 400 million relationships… that's quite a lot! The first thing these numbers tell us is that Go modules have 10 direct dependencies on average. For more interesting stats, let's write some Cypher, shall we? Indexing</h3> With this volume of data, the absolute first thing to do (that I clearly didn't do at first) is to create relevant indexes. I was initially under the impression that the module_identity</code> constraint previously created would also act as an index for :Module.name</code> since it's a composite unique index. I was wrong, and creating a specific index was necessary: CREATE INDEX module_name_idx IF NOT EXISTS FOR (m:Module) ON (m.name)</code></pre> I created other indexes as needed, and to do so the Cypher PROFILE</code></a> was a tremendous help. Find the direct dependents of a module</h3> As a warm-up, and to learn a bit more about Cypher, let's list the dependents of a specific module: MATCH (dependency:Module { name: 'github.com/pkg/errors', version: 'v0.9.1' }) MATCH (dependent:Module)-[:DEPENDS_ON]->(dependency) WHERE dependent.isLatest RETURN dependent.versionTime.year AS year, COUNT(dependent) AS nbDependents</code></pre> I chose github.com/pkg/errors@v0.9.1</code></a> because it's a module that was deprecated long ago, I find it interesting to know how much it's still used in the wild. Let's break the query down line by line: MATCH (dependency:Module { name: 'xxx', version: 'xxx' })</code> finds the module node (we know it's unique because of the constraint we declared earlier) with the given name</code> and version</code>. This is equivalent to: MATCH (dependency:Module) WHERE dependency.name = 'xxx' AND dependency.version = 'xxx'</code></pre></li> </ul> </li> MATCH (dependency)<-[:DEPENDS_ON]-(dependent:Module)</code> finds the module nodes with a direct DEPENDS_ON</code> relationship towards dependency</code>.</li> </ul> </li> WHERE dependent.isLatest</code> keeps dependents</code> modules that are in their latest version. This is useful because for any module x</code> depends on github.com/pkg/errors@v0.9.1</code>, we don't want to count all the versions of x</code> that depend on it. The latest is more relevant to us.</li> </ul> </li> RETURN dependent.versionTime.year AS year, COUNT(dependent) AS nbDependents</code> simply counts the total dependents of github.com/pkg/errors@v0.9.1</code> and group by release year of the dependent. If we wanted to list them, we could write: RETURN dependent.name AS dependentName ORDER BY dependentName;</code></pre></li> </ul> </li> </ol> Results: year</th> nbDependents</th></tr></thead> 2019</td> 3</td></tr> 2020</td> 6,774</td></tr> 2021</td> 10,680</td></tr> 2022</td> 11,747</td></tr> 2023</td> 8,992</td></tr> 2024</td> 12,220</td></tr> 2025</td> 16,001</td></tr> </tbody></table> That's a lot of dependents for a dead library! Find the transitive dependents of a module</h3> Neo4j really shines at graph traversal. Navigating relationships transitively requires virtually no changes: MATCH (dependency:Module { name: 'github.com/pkg/errors', version: 'v0.9.1' }) MATCH (dependent:Module)-[:DEPENDS_ON*1..]->(dependency) WHERE dependent.isLatest RETURN COUNT(dependent) AS nbDependents</code></pre> The only difference with the previous query is *1..</code>, asking Neo4j to follow the DEPENDS_ON</code> relationship transitively. We could also limit it to 2 levels with *1..2</code>. I find this interesting, because in the case of the query for direct dependencies, if we used a relational database, the SQL query would be pretty simple: SELECT COUNT(*) AS nb_dependents FROM dependencies d JOIN modules m ON m.id = d.dependent_id JOIN modules dependency ON dependency.id = d.dependency_id WHERE dependency.name = 'github.com/pkg/errors' AND dependency.version = 'v0.9.1' AND m.is_latest = true;</code></pre> but what's as simple as *1..</code> in Cypher would make a dramatically more complex SQL query: -- Example using a recursive CTE, I'm not sure every SGBDR implements it in the same way WITH RECURSIVE dependents_cte AS ( SELECT m.id AS dependency_id FROM modules m WHERE m.name = 'github.com/pkg/errors' AND m.version = 'v0.9.1' UNION ALL SELECT d.dependent_id FROM dependencies d JOIN dependents_cte cte ON d.dependency_id = cte.dependency_id ) SELECT COUNT(DISTINCT m.id) AS nb_dependents FROM modules m WHERE m.id IN (SELECT dependency_id FROM dependents_cte) AND m.is_latest = true;</code></pre>Top 10 most used dependencies</h3> Using the constructs from above, the query is once again pretty similar. MATCH (dependent:Module)-[:DEPENDS_ON]->(dependency:Module) WHERE dependent.isLatest RETURN dependency.name AS dependencyName, COUNT(dependent) AS nbDependents ORDER BY nbDependents DESC LIMIT 10;</code></pre> Results: dependencyName</th> nbDependents</th></tr></thead> github.com/stretchr/testify</td> 259,237</td></tr> github.com/google/uuid</td> 104,877</td></tr> golang.org/x/crypto</td> 100,633</td></tr> google.golang.org/grpc</td> 97,228</td></tr> github.com/spf13/cobra</td> 93,062</td></tr> github.com/pkg/errors</td> 92,491</td></tr> golang.org/x/net</td> 76,722</td></tr> google.golang.org/protobuf</td> 74,971</td></tr> github.com/sirupsen/logrus</td> 71,730</td></tr> github.com/spf13/viper</td> 64,174</td></tr> </tbody></table> github.com/stretchr/testify</code> is comfortably ahead of other dependencies as the most used in the open source Go ecosystem. Unsurprisingly, github.com/google/uuid</code> is also a staple library used pretty much everywhere. The golang.org/x/</code> dependencies also hold a strong place as the extended stdlib, as well as the infamous github.com/pkg/errors</code>. To get more insights, you can download the top 100 as a CSV file</a>. If you want to run your own queries, feel free to download my Neo4j dump via BitTorrent: </div> go-stats-neo4j-dump-20260105.tar</a> (11.21 GiB) </div> </div> To load it in a Neo4j instance, please follow these instructions: clone go-stats</code></a>,</li> copy the dump file to neo4j/backups/neo4j.dump</code> (the filename is important),</li> execute task neo4j:backup:load</code>, this will write the data to neo4j/data</code>,</li> run task neo4j:start</code> to start Neo4j.</li> </ul> You can then open localhost:7474</a> (no credentials needed) to use the Neo4j browser. That's all Folks!</h2> I hope you had a good time reading this post, and that you learned a thing or two! I'll probably continue playing around with this side-project as I have more ideas to explore. Specifically, I'd like to enrich the graph with more metadata, such as GitHub stars and tags. So stay tuned to the GitHub project</a> if you want to follow the upcoming developments. SQL Transactions in Go: The Good Way 2025-01-21T00:00:00+00:00 In my last work experience, I designed what I consider a pretty cool and efficient way to manage database transactions across a Go codebase. In a nutshell, I wanted to use database transactions in my business logic without exposing database internals. After dogfooding a solution for a while, I extracted it, improved it, and enriched it, to finally release it as the transactor</a> library. This library is now tested across the most common relational databases (PostgreSQL, MySQL, SQLite, MSSQL and Oracle), and is compatible with database/sql</code></a>, jmoiron/sqlx</a> and jackc/pgx</a>. In my current job, transactors are now our main (soon to be unique!) way to manage database transactions, which is why I feel comfortable talking about this method more widely. In this article, I’ll walk you through the design and implementation so you can either use the lib, or adapt it and extend it depending on your needs. Why exactly do we need a library?</h2> In web services / APIs, it’s customary to use some kind of layered architecture, like Clean Architecture, Hexagonal Architecture, or anything that boils down to a three-tier architecture. In Go, it’s something like: Handlers (a.k.a. controllers): the entrypoints of the API, usually the only layer allowed to be aware of the router and current HTTP request/response handlers,</li> Services (a.k.a. usecases): the business layer, where the domain logic lives,</li> Stores (a.k.a. repositories, data access, storage, persistence…): the layer communicating with databases, caches, filesystems, and so on. It’s usually the only layer allowed to be aware of the concrete storage systems. In practice, it means it’s the only place where database/sql</code>, your DB driver, or your ORM should be imported.</li> </ol> Whether transactions should be allowed in the services layer is sometimes up for debate, as they’re somewhat tied to the DB implementation. In Go, a transaction is represented by *sql.Tx</code>, but according to the definitions above, using it directly in the services layer would require importing database/sql</code>. This is why it’s sometimes accepted that transactions should live in the storage layer only. I disagree with this. I strongly believe transactions can be used both in the storage layer and in the services layer, for different reasons. Specifically, transactions can be a part of the business logic, and are a part of an interface contract. But I also believe neither *sql.DB</code>, *sql.Tx</code> or database/sql</code> should be imported by the services. This is the reason why I needed to create an object that could be injected to the services, allowing them to make transactions safely across many stores, without exposing any implementation details. The solution to this problem is the transactor</a> library. The Transactor interface</h2> A transactor lets you create a transactional context, represented by a closure. It’s defined as a simple interface: type Transactor interface { WithinTransaction(context.Context, func(ctx context.Context) error) error }</code></pre> A transactor implements this interface. We can then inject it in a service, and use it to make some transactional calls completely transparently, without knowing anything about the database. Let's look at some sample code. Example code without a transaction</h3> type service struct { balanceStore stores.Balance } func (s service) IncreaseBalance( ctx context.Context, account string, amount int, ) error { balance, err := s.balanceStore.GetBalance(ctx, account) if err != nil { return err } balance += amount err = s.balanceStore.SetBalance(ctx, account, balance) if err != nil { return err } return nil }</code></pre> This way to manage a financial balance is not great for several reasons, but let’s focus on it as a case study. Without going into too much details (“what transactions are used for” is not the topic here). This first version, doesn’t use a transaction. It makes it possible to get an inconsistent resulting balance. Example code using a transactor</h3> type service struct { balanceStore stores.Balance transactor Transactor } func (s service) IncreaseBalance( ctx context.Context, account string, amount int, ) error { return s.transactor.WithinTransaction(ctx, func(ctx context.Context) error { balance, err := s.balanceStore.GetBalance(ctx, account) if err != nil { return err } balance += amount err = s.balanceStore.SetBalance(ctx, account, balance) if err != nil { return err } return nil }) }</code></pre> This second version wraps GetBalance</code> and SetBalance</code> in a transaction, essentially making the operation atomic. The changes made when using the Transactor are minimal, as they just impact 2-3 lines in this case, but here’s what happens in details: WithinTransaction</code> accepts a context and a callback: The context will be enriched with the transaction ; more on that later,</li> The enriched context is passed to the callback, which I call a transactional context,</li> To keep the changes minimal, the parent context is shadowed in the callback by naming it ctx</code> ; this has no unintended effects, and makes it impossible to misuse the transactor (eg. by using the wrong context in one of the calls),</li> </ol> </li> WithinTransaction</code> does essentially three things to manage the transaction workflow: Begin the transaction,</li> Execute the callback,</li> Commit, or Rollback the transaction.</li> </ol> </li> </ol> This workflow is detailed in the flowchart below. flowchart TD Start((Start)) --> Begin["Begin transaction"] Begin --> ErrBegin{Error?} ErrBegin -- no --> Callback["Execute callback"] ErrBegin -- yes --> RetErrBegin([Return error]) Callback --> ErrCallback{Error?} ErrCallback -- no --> Commit ErrCallback -- yes --> RollbackCallback["Rollback"] RollbackCallback --> RetErrCallback([Return error]) Commit --> ErrCommit{Error?} ErrCommit -- no --> Success([Success]) ErrCommit -- yes --> RetErrCommit([Return error]) classDef failure stroke:#eb2f06,fill:#e55039 classDef success stroke:#079992,fill:#38ada9 class RetErrBegin,RollbackCallback,RetErrCallback,RetErrCommit failure class Success success </pre> And voilà! The transactor makes using transactions fool proof, and extremely easy. And the best thing is, there’s more. Nesting transactions</h2> Transactors, when properly implemented, make it possible to deal with nested transactions</a>. The implementation for nested transactions varies depending on the database, but the transactor library provides an implementation for all the major systems. You can use a transactor anywhere, and you don’t have to worry if the methods you call use a transactor themselves. As an example, the following code making use of our previously defined IncreaseBalance</code> is completely valid and working as you would expect: func (s service) TransferBalance( ctx context.Context, fromAccount, toAccount string, amount int, ) error { return s.transactor.WithinTransaction(ctx, func(ctx context.Context) error { err := s.DecreaseBalance(ctx, fromAccount, amount) if err != nil { return err } err = s.IncreaseBalance(ctx, toAccount, amount) if err != nil { return err } return nil }) }</code></pre> or, more generally: if err := s.transactor.WithinTransaction(ctx, func(ctx context.Context) error { // Do some stuff before the nested transaction if err := s.transactor.WithinTransaction(ctx, func(ctx context.Context) error { // Do some stuff inside the nested transaction }; err != nil { // Handle rollback of the nested transaction } // Do some stuff after the nested transaction }; err != nil { // Handle rollback of the main transaction }</code></pre> Just like that, we made transactional methods composable inside a larger transaction. Compared to other ways of dealing with transactions, such as Unit of Work</a>, transactors make it really straightforward to compose anything inside a transaction: store methods, methods across different stores (working on the same storage system), or business services. Implementing a database/sql</code> Transactor</h2> To illustrate the above, let’s dive into the implementation of a transactor for database/sql</code> from the Go standard library. The full reference implementation can be found in transactor/stdlib/transactor.go</a>. The very basic structure we’ll need will evolve a bit as we go, but let’s get started: package transactor import ( "context" "database/sql" ) type Transactor interface { WithinTransaction(context.Context, func(ctx context.Context) error) error } type transactor struct{ db *sql.DB } var _ Transactor = &transactor{} func NewTransactor(db *sql.DB) *transactor { return &transactor{db: db} } func (t *transactor) WithinTransaction(ctx context.Context, txFunc func(context.Context) error) error { return nil }</code></pre> In this first step, we create a private transactor</code> implementing the Transactor</code> interface. *sql.DB</code> is a database handler representing an active connection to the database. It’s the result of sql.Open</code></a>. With the general structure in place, we can start implementing the WithinTransaction</code> logic. As we expect it, its role will be to manage the transaction and execute the provided callback within this context. func (t *transactor) WithinTransaction(ctx context.Context, txFunc func(context.Context) error) error { tx, err := t.db.BeginTx(ctx, nil) if err != nil { return fmt.Errorf("failed to begin transaction: %w", err) } txCtx := txToContext(ctx, tx) if err := txFunc(txCtx); err != nil { _ = tx.Rollback() return err } if err := tx.Commit(); err != nil { return fmt.Errorf("failed to commit transaction: %w", err) } return nil }</code></pre> This implementation already deals with most of the workflow: the transaction is started with BeginTx</code>, the txFunc</code> callback is then executed, and the transaction is either Commit</code>ted or Rollback</code>ed depending on the callback result. The actual magic happens at line 7, with the txToContext</code> function call: txToContext</code> stores the transaction (an *sql.Tx</code> instance) as a context key,</li> the context, enriched with the transaction, is passed to the callback function.</li> </ul> We can implement txToContext</code> like this: type txCtxKey struct{} func txToContext(ctx context.Context, tx pgx.Tx) context.Context { return context.WithValue(ctx, txCtxKey{}, tx) }</code></pre> Using a private struct type as a context key ensures nothing will conflict: this key can only be set by using txCtxKey</code>. At this point, the transactor logic is done. The actual transactor/stdlib</code></a> implementation is a bit more complex because it can deal with nested transactions, but the logic is the same. The only thing we miss is a way to actually use our transactor. When making our database queries, we must either check if our context contains a transaction, or use the DB connection directly. To do so, we need a helper to give us the current transaction or connection from a context: the DBGetter</code>. Let’s modify the NewTransactor</code> constructor to give us a DBGetter</code>: type DBGetter func(context.Context) DB func NewTransactor(db *sql.DB) (*transactor, DBGetter) { return &transactor{db: db}, func(ctx context.Context) DB { if tx := txFromContext(ctx); tx != nil { return tx } return db } }</code></pre> txFromContext</code> does the opposite of our previous txToContext</code> function and is implemented as: func txFromContext(ctx context.Context) pgx.Tx { tx, ok := ctx.Value(txCtxKey{}).(pgx.Tx) if ok { return tx } return nil }</code></pre> DB</code> is simply an interface defining the common methods between *sql.DB</code></a> and *sql.Tx</code></a>, so that we can use a DB handler or a transaction indistinctly: type DB interface { ExecContext(ctx context.Context, query string, args ...any) (sql.Result, error) PrepareContext(ctx context.Context, query string) (*sql.Stmt, error) QueryContext(ctx context.Context, query string, args ...any) (*sql.Rows, error) QueryRowContext(ctx context.Context, query string, args ...any) *sql.Row }</code></pre> The DBGetter</code> we define will always return the initial *sql.DB</code> handler by default, even if the context is completely empty (eg. context.Background()</code> or nil</code>), which makes it safe to use anywhere. The only way it can return a transaction is if it’s called inside a WithinTransaction</code> block, as part of the txFunc</code> callback we defined earlier. These changes conclude the transactor implementation. As you’ll see in the examples below, the key for it to work as intended is to always use the DBGetter</code> to make database queries, never the *sql.DB</code> directly. This is the only way that these functions will know to use an active transaction. The following code is an example implementation of the balanceStore</code> described at the beginning of this article, making use of the DBGetter</code>: type balanceStore struct { dbGetter DBGetter } func (s balanceStore) GetBalance(ctx context.Context, account string) (int, error) { var amount int err := s.dbGetter(ctx).QueryRow(ctx, "SELECT amount FROM balances WHERE id = $1 FOR UPDATE", account).Scan(&amount) return amount, err } func (s balanceStore) SetBalance(ctx context.Context, account string, balance int) error { _, err := s.dbGetter(ctx).Exec(ctx, "UPDATE balances SET amount = $1 WHERE id = $2", amount, account) return err } func main() { ctx := context.Background() db, _ := sql.Open("<driver>", "<dsn>") transactor, dbGetter := transactor.NewTransactor(db) balanceStore := balanceStore{dbGetter: dbGetter} err := transactor.WithinTransaction(ctx, func(ctx context.Context) error { balance, err := balanceStore.GetBalance(ctx, "account-1") if err != nil { return err } balance += 10 err = s.balanceStore.SetBalance(ctx, "account-1", balance) if err != nil { return err } return nil }) if err != nil { // ❌ Transaction rollbacked return } // ✅ Transaction committed }</code></pre> Using the context to pass information is sometimes controversial in the Go community, but in this case I feel the benefits vastly outweigh the cons. In fact, after using transactors for almost a full year in various situations, I'd say there isn't anything that would make me go back to an alternative method for managing transactions. Anyway, that’s it for today! I hoped you enjoyed this article, and if you did, I encourage you to give my transactor</a> library a try in one of your projects, and maybe ⭐ it on GitHub</a>! Writing & Testing a Paginated API Iterator in Go 2024-12-19T00:00:00+00:00 Go 1.23, amongst other features</a>, brought Iterators</a> to the standard library. Iterators are basically a way to make the range</code> operator work on functions implementing a specific interface. This lets you write code like this: for value := range myIterator() { // ... }</code></pre> The main difference with using range</code> over a slice or a map is that an iterator can fetch data lazily. I doesn’t need to fetch all the data at once before writing the loop. This comes really handy for consuming paginated APIs: we can write an iterator abstracting the pagination logic, making the code more readable and reusable. In this article I’ll show you how to write a custom iterator for consuming a paginated HTTP API (we’ll use GitHub’s API as an example), and how to test it properly by leveraging Pull Iterators</a>. Note that this article will not discuss the design of iterators in Go, we’ll just see how to use and work with them. For more information on iterators, I recommend these articles: Ranging over functions in Go 1.23</a></li> Range Over Function Types</a></li> </ul> Writing an Iterator</h2> Iterators can return up to two values. They must implement one of these signatures: iter.Seq</code></a> (func(yield func(V) bool</code>): iterators over sequences of 1 value</li> iter.Seq2</code></a> (func(yield func(K, V) bool</code>): iterators over sequences of 2 values</li> func(yield func() bool)</code>: iterators over sequence of… no value :)</li> </ul> For iterators working over remote resources like APIs, the emerging idiom is to use the second value for error handling. In practice, using our iterator will look like this: // IterateResources is a function returning an iter.Seq2[K, error] for resource, err := range IterateResources(ctx, apiClient) { if err != nil { // handle error break } // use resource }</code></pre> I noticed a few naming conventions for functions returning an iterator: IterateResources</code></li> IterResources</code></li> ResourcesSeq</code></li> </ul> I’m not sure yet which is the more common, but I went with Iterate</code> in this article as it feels clearer. We’ll implement an iterator looking like the above using the GitHub API to list the repositories of a user</a>. I chose this API for 3 reasons: it's public and doesn't require authentication. You can give it a try in a terminal:curl 'https://api.github.com/users/thiht/repos?page=1&per_page=5'</code></pre></li> it's paginated, so it's a good use-case for an iterator,</li> the Go module google/go-github</a> is readily available to interact with the GitHub API.</li> </ol> With google/go-github</a>, we’ll use the RepositoriesService.ListByUser</a> method. As a starting point, let’s write a version of the code listing all my repositories</a>, without an iterator. package main import ( "context" "github.com/google/go-github/v67/github" ) func main() { ctx := context.Background() client := github.NewClient(nil) user := "thiht" opts := &github.RepositoryListByUserOptions{ ListOptions: github.ListOptions{ Page: 1, PerPage: 5, }, } for { repos, resp, err := client.Repositories.ListByUser(ctx, user, opts) if err != nil { panic(err) } for _, repo := range repos { println(*repo.FullName) } if resp.NextPage == 0 { break } opts.Page = resp.NextPage } }</code></pre> The pagination parameters are passed via ListOptions</code>, and the next page to fetch is retrieved at each loop from the resp</code> result. I set the pagination settings to list the repositories 5 by 5 so that we get a chance to see the pagination in action. This code works fine, but it forces us to mix the pagination logic (checking the NextPage</code> value and updating opts.Page</code>) with our business logic (printing the repository name). It also suffers from a lack of standardization: any API you use will be different. Not only REST APIs, but also filesystem APIs, custom collections, or whatever. Iterators will help us abstract anything related to the pagination and focus on our own logic. Let’s now rewrite the same code with an iterator: package main import ( "context" "iter" "github.com/google/go-github/v67/github" ) // 1. Iterator usage func main() { ctx := context.Background() client := github.NewClient(nil) user := "thiht" for repo, err := range IterateRepositoriesByUser( ctx, client.Repositories, user, nil, ) { if err != nil { panic(err) } println(*repo.FullName) } } // 2. Iterator implementation func IterateRepositoriesByUser( ctx context.Context, client RepositoryLister, user string, opts *github.RepositoryListByUserOptions, ) iter.Seq2[*github.Repository, error] { return func(yield func(*github.Repository, error) bool) { // 2.1. Initialization if opts == nil { opts = &github.RepositoryListByUserOptions{} } if opts.Page == 0 { opts.Page = 1 } if opts.PerPage == 0 { opts.PerPage = 5 } // 2.2. Pagination loop for { repos, response, err := client.ListByUser(ctx, user, opts) if err != nil { yield(nil, err) return } for _, repo := range repos { if !yield(repo, nil) { return } } if response.NextPage == 0 { return } opts.Page = response.NextPage } } } type RepositoryLister interface { ListByUser( ctx context.Context, user string, opts *github.RepositoryListByUserOptions, ) ([]*github.Repository, *github.Response, error) }</code></pre> As you can see, the iterator version is slightly more verbose, but it's more reusable and has the benefit of isolating the pagination logic from the rest of the code. Keep in mind in real life projects you’ll modify the business logic of the loop more often than you’ll modify the iterator itself, so it makes sense to take a bit of time writing an iterator to get simpler business code. IterateRepositoriesByUser</code> is not an iterator itself, but a function returning an iterator. This lets us pass the parameters that will be available in the scope of the iterator: the context, the GitHub client, the target user, and the additional options. I declared a RepositoryLister</code> interface to make it easier to test the iterator, we’ll come back to it in the next part. The iterator consists of these steps: 2.1: Initialization of the pagination params before calling the API: the pagination starts at page 1 on the GitHub API,</li> 2.2: Main loop listing the user’s repositories. Note that this is roughly the same code we had in the implementation without iterator: the pagination logic stays the same. The main difference is the use of yield</code> to return our values and errors to range</code>: If ListByUser</code> fails, yield the error and return: I decided to return unconditionally in case of failure, because I want the range</code> to end if ListByUser</code> fails.</li> If ListByUser</code> succeeds, the repositories are yielded one by one.</li> </ul> </li> </ul> As a reminder: yield</code> returns true</code> if the range</code> loop continues (if the continue</code> keyword is used, or if the loop continues to the next iteration),</li> yield</code> returns false</code> if the range</code> loop stops (if break</code> or return</code> are used, or if the loop is interrupted).</li> </ul> Testing the Iterator</h2> It can be a bit challenging to write tests for an iterator. In our tests we need to make sure that: the iterator contains the elements we expect, in the correct order,</li> the iterator doesn’t contain more elements than we expect.</li> </ul> Mocking RepositoryLister</code></h3> Using a mock is not the only way to write this test, just my personal preference. As github.NewClient</code> accepts an *http.Client</code> as a parameter, we could instead use net/http/httptest</code></a> to control the HTTP calls and their responses. In the above example, we used an interface RepositoryLister</code> to abstract the GitHub API client. Thanks to this abstraction, we can mock the GitHub API client in our tests. We could use a mocking library like mockery</a> to generate a mock for this interface automatically, but I'll write it manually for this example: import ( "context" "fmt" "testing" "github.com/google/go-github/v67/github" ) type repositoryListerMock struct { t *testing.T calls []callListByUser } type callListByUser struct { expectPage int repos []*github.Repository resp *github.Response err error } func newRepositoryListerMock(t *testing.T, calls ...callListByUser) *repositoryListerMock { mock := repositoryListerMock{t: t, calls: calls} t.Cleanup(func() { if len(mock.calls) > 0 { t.Fatalf("%d unfulfilled calls to ListByUser", len(mock.calls)) } }) return &mock } func (r *repositoryListerMock) ListByUser(ctx context.Context, user string, opts *github.RepositoryListByUserOptions) ([]*github.Repository, *github.Response, error) { r.t.Helper() if len(r.calls) == 0 { r.t.Fatal("no result registered for ListByUser") } if opts.Page != r.calls[0].expectPage { r.t.Fatalf("unexpected page: got %d, want %d", opts.Page, r.calls[0].expectPage) } result := r.calls[0] r.calls = r.calls[1:] return result.repos, result.resp, result.err }</code></pre> This mock lets you register the expected calls to ListByUser</code> and the results you want to return. It can be used as follows, in place of a RepositoryLister</code>: func Test_repositoryListerMock(t *testing.T) { mockClient := newRepositoryListerMock(t, callListByUser{ // First call expectPage: 1, repos: []*github.Repository{ {FullName: github.String("example")}, }, resp: &github.Response{NextPage: 2}, }, callListByUser{ // Second call expectPage: 2, err: fmt.Errorf("an error occurred"), }, ) repos, resp, err := mockClient.ListByUser(context.Background(), "thiht", &github.RepositoryListByUserOptions{ ListOptions: github.ListOptions{Page: 1}, }) if err != nil { t.Errorf("unexpected error %v:", err) } if len(repos) != 1 { t.Errorf("unexpected number of repositories: got %d, want %d", len(repos), 1) } if *repos[0].FullName != "example" { t.Errorf("unexpected repository name: got %q, want %q", *repos[0].FullName, "example") } if resp.NextPage != 2 { t.Errorf("unexpected next page: got %d, want %d", resp.NextPage, 2) } _, _, err = mockClient.ListByUser(context.Background(), "thiht", &github.RepositoryListByUserOptions{ ListOptions: github.ListOptions{Page: 2}, }) if err == nil { t.Errorf("expected error, got nil") } // An additional call to ListByUser will fail because there are no more results // mockClient.ListByUser(context.Background(), "thiht", nil) }</code></pre>Writing tests for the sequence</h3> To test an iterator, I find it easier to first write down the expected sequence of values it returns. To do so, we can start by defining a type to represent the 2 values returned by an iteration: type iteration struct { repo *github.Repository err error }</code></pre> This way we’ll be able to declare an expected sequence for a test: expectedIterations := []iteration{ {repo: &github.Repository{FullName: github.String("example1")}}, {repo: &github.Repository{FullName: github.String("example2")}}, }</code></pre> Before writing our test, let’s setup our GitHub client mock with the expected responses to match this expected sequence: client := &repositoryListerMock{ t: t, calls: []callListByUser{ { expectPage: 1, repos: []*github.Repository{ {FullName: github.String("example1")}, {FullName: github.String("example2")}, }, resp: &github.Response{NextPage: 0}, }, }, }</code></pre> Our repositoryListerMock</code> implements ListByUser, will return 2 repositories on the first page, and signal there’s no next page to fetch. With all of this setup, we can finally test the iterator itself. The first trick here is to convert our iterator to a pull iterator to get finer control over the iteration sequence: // 1. Initialize the iterator next, stop := iter.Pull2(IterateRepositoriesByUser(context.Background(), client, "thiht", nil)) t.Cleanup(stop) // 2. Iterate and compare to the expected sequence for _, expected := range expectedIterations { repo, err, ok := next() if !ok { t.Fatal("unexpected end of iteration") } if !reflect.DeepEqual(repo, expected.repo) { t.Errorf("unexpected repository: got %+v, want %+v", repo, expected.repo) } if err != expected.err { t.Errorf("unexpected error: got %v, want %v", err, expected.err) } } // 3. Ensure the iterator is empty if _, _, ok := next(); ok { t.Fatal("unexpected iteration") }</code></pre> The actual testing consists of 3 parts: First, the iterator is initialized by calling IterateRepositoriesByUser</code> with its expected parameters, including the mock client we created previously. We call iter.Pull2</code></a> to convert the iterator to a push iterator, which gives us 2 new variables: next</code> and stop</code>. next</code> will give us the next values in the iterator, and a boolean indicating whether it still contained values.</li> Then, we loop on the expectedIterations</code>. That’s the second trick, we don’t loop directly on the iterator but instead on the sequence we expect. This lets us check all the values we expect one by one. The first action in the loop is to call next</code> to iterate. If the ok</code> value returned by the iterator is false</code> it means we expected more values than the iterator gave us, so the test fails.</li> Then, we can simply check that the returned *github.Repository</code> and error</code> are the ones we expected.</li> </ul> </li> Finally we need to call next</code> one last time to ensure the iterator is empty. If the ok</code> value is true</code>, it means the iterator contains more values than we expected. The test fails, and we need to rework our expectedSequence</code>.</li> </ol> This approach makes it easy to test all the possible cases for this iterator. The below examples show how to write a few additional tests, such as an iterator calling multiple pages of results, or an iterator returning an unexpected error. func TestIterateRepositoriesByUser(t *testing.T) { type iteration struct { repo *github.Repository err error } t.Run("1 page of repositories", func(t *testing.T) { client := &repositoryListerMock{ t: t, calls: []callListByUser{ { expectPage: 1, repos: []*github.Repository{ {FullName: github.String("example1")}, {FullName: github.String("example2")}, }, resp: &github.Response{NextPage: 0}, }, }, } expectedIterations := []iteration{ {repo: &github.Repository{FullName: github.String("example1")}}, {repo: &github.Repository{FullName: github.String("example2")}}, } next, stop := iter.Pull2(IterateRepositoriesByUser(context.Background(), client, "thiht", nil)) t.Cleanup(stop) for _, expected := range expectedIterations { repo, err, ok := next() if !ok { t.Fatal("unexpected end of iteration") } if !reflect.DeepEqual(repo, expected.repo) { t.Errorf("unexpected repository: got %+v, want %+v", repo, expected.repo) } if err != expected.err { t.Errorf("unexpected error: got %v, want %v", err, expected.err) } } if _, _, ok := next(); ok { t.Fatal("unexpected iteration") } }) t.Run("2 pages of repositories", func(t *testing.T) { client := &repositoryListerMock{ t: t, calls: []callListByUser{ { expectPage: 1, repos: []*github.Repository{ {FullName: github.String("example1")}, }, resp: &github.Response{NextPage: 2}, }, { expectPage: 2, repos: []*github.Repository{ {FullName: github.String("example2")}, }, resp: &github.Response{NextPage: 0}, }, }, } expectedIterations := []iteration{ {repo: &github.Repository{FullName: github.String("example1")}}, {repo: &github.Repository{FullName: github.String("example2")}}, } next, stop := iter.Pull2(IterateRepositoriesByUser(context.Background(), client, "thiht", nil)) t.Cleanup(stop) for _, expected := range expectedIterations { repo, err, ok := next() if !ok { t.Fatal("unexpected end of iteration") } if !reflect.DeepEqual(repo, expected.repo) { t.Errorf("unexpected repository: got %+v, want %+v", repo, expected.repo) } if err != expected.err { t.Errorf("unexpected error: got %v, want %v", err, expected.err) } } if _, _, ok := next(); ok { t.Fatal("unexpected iteration") } }) t.Run("api error", func(t *testing.T) { clientErr := fmt.Errorf("an error occurred") client := &repositoryListerMock{ t: t, calls: []callListByUser{ { expectPage: 1, err: clientErr, }, }, } expectedIterations := []iteration{ {err: clientErr}, } next, stop := iter.Pull2(IterateRepositoriesByUser(context.Background(), client, "thiht", nil)) t.Cleanup(stop) for _, expected := range expectedIterations { repo, err, ok := next() if !ok { t.Fatal("unexpected end of iteration") } if !reflect.DeepEqual(repo, expected.repo) { t.Errorf("unexpected repository: got %+v, want %+v", repo, expected.repo) } if err != expected.err { t.Errorf("unexpected error: got %v, want %v", err, expected.err) } } if _, _, ok := next(); ok { t.Fatal("unexpected iteration") } }) t.Run("no repositories", func(t *testing.T) { client := &repositoryListerMock{ t: t, calls: []callListByUser{ { expectPage: 1, repos: []*github.Repository{}, resp: &github.Response{NextPage: 0}, }, }, } next, stop := iter.Pull2(IterateRepositoriesByUser(context.Background(), client, "thiht", nil)) t.Cleanup(stop) if _, _, ok := next(); ok { t.Fatal("unexpected iteration") } }) }</code></pre> TDD is a personal practice 2023-12-18T00:00:00+00:00 I know TDD. I know what real TDD is. I know how to do it right, whatever that means depending on the TDD practitioner. I still don’t like this practice, and rarely use it, except in 2 situations: I’m fixing a bug in an existing code base: in this case I’ll write a test first to reproduce and isolate the bug, and then fix it and refactor if needed</li> I’m adding a new use case to an existing feature, and will treat the lack of existence of this new case as a bug, see 1.</li> </ol> I don’t ever, ever, EVER use TDD to write new features from scratch. It doesn’t work for me. It doesn’t match my mental model. It doesn’t help me go faster, doesn’t make me produce better code, and doesn’t make me write more, or more useful tests. I don't enjoy it and it doesn't provide me with any kind of satisfaction. I still have a personal methodology, but it’s not TDD. I still strive to write good test and maintain good coverage, but not doing TDD. TDD should not be forced upon developers who don’t want to follow it, in the same way a specific IDE should not be forced upon a developer. TDD is a tool, not a goal in itself. The shared goal is to have a well tested code base that is as safe as possible from regressions. How developers achieve that doesn’t matter. TDD should not be conflated as "writing tests". I often see the rhetoric that, if you don’t do TDD, then you don’t write tests. If I showed you a code base with tests, there would be no way for you to know if they’ve been written in TDD or not. TDD is not a team practice. Writing tests is a team practice. Code review is a team practice. Automated CI is a team practice. Whether a developer does or does not do TDD has no impact on the rest of the team. I could tell you I'm doing TDD if it pleases you, and you'd have no way to know that I don't, micro-managing set aside. If in TDD you trust, don’t proselytize it. TDD is a personal practice. Shell Scripts Matter 2017-12-03T00:00:00+00:00 The shell is an odd beast. Although it goes against every current trend in software engineering (strong typing, compile checks over runtime checks, ...), shell scripts are here to stay, and still constitute an important part of every developer's life. The weird thing about shell scripts is that even strong advocates of good practices gladly forget all they know when it comes to shell scripting. Versioning? Why bother, it's disposable code. Code quality? That's just a shell script, it's garbage anyway. Testing? Nah. There aren't any decent tools for that. </blockquote> Wrong, wrong, and wrong. Shell scripts have value. Everything you do for real code should be done for non trivial shell scripts, even for a one-time script. That includes versioning, code reviews, continuous integration, static code analysis, and testing. Here is a summary of everything that can, and should be done when writing shell scripts. Note: This article will use Bash as a reference shell. Most of the content can be transposed to other POSIX compliant shells. Keep your scripts in version control</h2> Keeping shell scripts under version control has multiple advantages: It constitutes a library. Shell scripts can be hard to write. If there's a reference for something difficult somewhere, your coworkers will thank you when they need it. You should setup a "shell-scripts" repository somewhere as soon as possible.</li> They can be properly reviewed. Making mistakes is easy with shell scripts, and they can be very damaging. Code review should be mandatory for shell scripts, as for any other piece of code.</li> They can be improved. I won't explain to you what version control is. But with shell scripts versioned, it's easy to improve them regularly.</li> </ul> Please, from now on, version all your shell scripts before running them. Have someone reviewing your scripts in priority before executing them in production. It's not a waste of your coworkers' time, it's a time saver for the team. Improve the quality of your scripts with ShellCheck</h2> Although you can check the syntactic validity of your scripts with the command bash -n</code>, much powerful tools exist. ShellCheck</a> is a static code analysis tool for shell scripts. It's really an awesome tool which will help you improve your skills as you use it. So do use it. You can install it globally on your machine</a>, use it in your continuous integration</a>, and it even integrates perfectly with most major editors</a>. There really are no downsides to using ShellCheck and it can save you from yourself. If Steam had used ShellCheck in 2015, this line would never have made it to production</a>: rm -rf "$STEAMROOT/"*</code></pre> This code violates the SC2115 rule</a> from ShellCheck. Use Bash unofficial strict mode</h2> The unofficial strict mode comes from Aaron Maxwell's article "Use the Unofficial Bash Strict Mode (Unless You Looove Debugging)</a>". He suggests to start every Bash script with the following lines: #!/bin/bash set -euo pipefail IFS=$'\n\t'</code></pre> set -e</code> will exit the script if any command returns a non-zero status code. To prevent the option from triggering on commands returning a non-zero status code even when no error occurred, there are two solutions: using the || true</code> pattern: command_returning_non_zero || true</code></pre></li> temporary disabling the option: set +e command_returning_non_zero set -e</code></pre></li> </ul> </li> set -u</code> will prevent using an undefined variable. In the case of undefined positional parameters ($1</code>, $2</code>, ...), you can give them a default value with the parameter expansion</a> construct: my_arg=${1:-"default"}</code></pre></li> set -o pipefail</code> will force pipelines to fail on the first non-zero status code. </li> IFS=$'\n\t'</code> makes iterations and splitting less surprising, in the case of loops mostly. The default for this variable is usually IFS=$' \n\t'</code> but the space as a separator often gives confusing results. </li> </ul> Read the original article</a> for more details and solutions for common challenges when using the strict mode</a>! The unofficial strict mode is more intrusive than what we've seen before and can be hard to deal with, but it's worth it in the long run. Take the time to try it. Do some cleanup!</h2> When scripts are interrupted, either because of a user's action or because something bad occurred, most shell scripts don't clean up their mess. In the worst case, they might not restart services they had to temporarily disable. It's a shame given how easy it is to perform some cleanup and error catching with the trap</code> command. Once again, in "How "Exit Traps" Can Make Your Bash Scripts Way More Robust And Reliable</a>", Aaron Maxwell gives some great advice. Always add the following in your shell scripts: cleanup() { # ... } trap cleanup EXIT</code></pre> The trap</code> command will execute the cleanup</code> function as soon as the script exits. In this function you could remove temporary files, restart services, or whatever is relevant to your script. Test your scripts with shUnit2</h2> shUnit2</a> is a unit testing framework for shell scripts. It's inspired by JUnit. It's available in the standard repositories so you can install it with apt-get install shunit2</code> on an Ubuntu-based distro. shUnit2 consists of a shell script you can source</code> in your test file. To use it, there are multiple approaches. In order not to clutter the main script, I prefer writing the tests in a separate file. This means I'll have a script.sh</code> file and a test_script.sh</code> file. Below is an example for a script offering a function to add two numbers. add.sh</code> must have the following structure: add() { local a=$1 local b=$2 echo $(( a + b )) } if [[ "${BASH_SOURCE[0]}" = "$0" ]]; then # Main code of the script add $1 $2 fi</code></pre> The [[ "${BASH_SOURCE[0]}" = "$0" ]]</code> test is used to execute the main code only when the script is executed directly, not source</code>d. test_add.sh</code> will look like this: . ./add.sh test_add() { actual=$(add 5 8) expected=13 assertEquals "$expected" "$actual" } . shunit2</code></pre> First, the test file source</code>s the main file add.sh</code> (in Bash, .</code> is an alias to source</code>). The functions it declares are then available in the test script. The actual tests are simple functions with a name starting by test</code>. At the end, the globally installed shunit2</code> is source</code>d and performs its magic</a>. The test file can then be executed: $ bash test_add.sh test_add Ran 1 test. OK</code></pre> The details of what shUnit2 can do are explained in its documentation</a>. There are alternatives to shUnit2, such as Bats</a> or Roundup</a> but I didn't have a chance to use them yet. Their usage should be relatively similar though. The point of this section is that testing shell scripts is doable and should be done, whatever solution you choose in the end. Log what your script is doing</h2> In the past, I made the mistake of not logging anything. I liked running a script and seeing it work magically without anything ugly showing in the console. I was wrong, because when something doesn't work as expected, it becomes impossible to know what happened. Running a script is not supposed to feel like magic, it must be somewhat verbose and understandable. For that, please log as much as possible in your scripts. For this purpose, I usually add the following lines in my scripts: readonly LOG_FILE="/tmp/$(basename "$0").log" info() { echo "[INFO] $*" | tee -a "$LOG_FILE" >&2 ; } warning() { echo "[WARNING] $*" | tee -a "$LOG_FILE" >&2 ; } error() { echo "[ERROR] $*" | tee -a "$LOG_FILE" >&2 ; } fatal() { echo "[FATAL] $*" | tee -a "$LOG_FILE" >&2 ; exit 1 ; }</code></pre> This tiny logging framework allows to easily keep track of whatever happens during the script execution. Logging becomes as simple as writing info "Executing this and that..."</code>. Then it's easy to grep</code> on the log file to find something specific. Feel free to improve these functions as you need, with the date, the calling function name (with $FUNCNAME</code>), etc. I don't use the builtin logger</code></a> because it requires special privileges to write to /var/log</code> and I'm not fond of its usage. Writing to a log file in /tmp</code> is usually good enough. For cron</code> scripts though you should probably investigate logger</code>. Use the -v</code> or --verbose</code> of the commands you invoke as needed to improve the quality of your logging. Learn to debug your scripts</h2> The easiest way to debug a shell script, besides logging, is to run it with bash -x</code>. Another way is to use set -x</code> inside the script. This option will make Bash print every command before its execution, replacing the variables with their real values. Used together with the unofficial strict mode, this method is useful to see what's going on in a script with less risk to break the environment. It's also worth knowing that a few debuggers for Bash exist, for example bashdb</a>. bashdb works in the same way as gdb, and can be used to add breakpoints, switching to step by step execution, showing the value of variables, etc. You can learn how to use bashdb with the video "Using BashDB to Debug Your Shell Scripts </a>": </iframe> </div> Document your scripts</h2> Any shell script should have a --help</code> option. This doesn't seem easy? It is, thanks to the following wizardry: #/ Usage: add <first number> <second number> #/ Compute the sum of two numbers usage() { grep '^#/' "$0" | cut -c4- exit 0 } expr "$*" : ".*--help" > /dev/null && usage</code></pre> The usage</code> function will print every line starting with #/</code> comments, without this prefix. The expr</code> command will check if the string resulting of the concatenation of all the parameters contains --help</code>. If so, it will call usage</code>. This is definitely not the cleanest way to parse parameters, but this quick method ensures you will add a --help</code> flag. For the sake of good practices, this StackOverflow post</a> explains how to properly parse a script's parameters using the while</code>/case</code>/shift</code> construct. To generate HTML documentation from your comments, you can also check shocco.sh</a>, which inspired the above trick</a>. Random advice</h2> The following is a list of random good practices I've learned the hard way. I'll explain the rationale behind every advice as I go. Use Bash for scripting</h3> Use Bash by default, Sh if you have to. Try to forget about Ksh, Zsh or Fish unless there are really good reasons to use them. This choice not only ensures your script will work virtually everywhere, but also fosters comprehension of a script by the whole team. You don't write production scripts for yourself. Use Bashisms</h3> If you use Bash, don't half-use it. Use parameter expansion</a>. Use local</a> and readonly</a> variables. Use improved conditional expressions</a>. Quote your variables</h3> Even if you know quoting is not required, it's a good habit to always quote variables. The only exception is when you specifically want expansion to occur. More on word splitting</a>. Name your parameters</h3> This goes without saying, but explicitly naming parameters ($1</code>, $2</code>, ...) makes the code self-documenting and helps readability. The parameter expansion of Bash is a great candidate for naming and assigning default values to positional parameters: my_arg=${1:-default}</code></pre>Use subshells as a way to control what's in your global scope</h3> An example is worth a thousand words: var=1 echo $var ( echo $var var=5 echo $var ) echo $var</code></pre> will print: 1 1 5 1</code></pre> My main usage for this is when I need to temporarily modify $IFS</code> (for iterating over simple CSV-like files for example) and reset it to its original value afterwards. Use a template</h3> This script template summarizes every snippets shared along this article. I believe it's a good basis for any kind of script. #!/usr/bin/env bash set -euo pipefail IFS=$'\n\t' #/ Usage: #/ Description: #/ Examples: #/ Options: #/ --help: Display this help message usage() { grep '^#/' "$0" | cut -c4- ; exit 0 ; } expr "$*" : ".*--help" > /dev/null && usage readonly LOG_FILE="/tmp/$(basename "$0").log" info() { echo "[INFO] $*" | tee -a "$LOG_FILE" >&2 ; } warning() { echo "[WARNING] $*" | tee -a "$LOG_FILE" >&2 ; } error() { echo "[ERROR] $*" | tee -a "$LOG_FILE" >&2 ; } fatal() { echo "[FATAL] $*" | tee -a "$LOG_FILE" >&2 ; exit 1 ; } cleanup() { # Remove temporary files # Restart services # ... } if [[ "${BASH_SOURCE[0]}" = "$0" ]]; then trap cleanup EXIT # Script goes here # ... fi</code></pre>Stay informed</h3> Shell scripting isn't moving that much these days. It's a great reason to read stuff about the topic, it makes it easy to keep up! Here are some interesting resources: man bash</code>, I bet almost none of you have read it, yet it's a great source of information! And it's not that hard to read, I promise.</li> @UnixTooltip</a> on Twitter, continuously gives tips and tricks.</li> Most upvoted Bash questions on StackOverflow</a></li> Bash participative documentation on StackOverflow</a> has some interesting examples.</li> ShellCheck's wiki</a> is a good resource to learn what not to do, and why.</li> Hacker News</a> and /r/programming</a>, once in a while articles on this subject pop out.</li> </ul> I hope this article brought light on what was possible with shell scripting. The tools are there. You know the practices. Now it's up to you to make scripting a delight to work with, for you and your team! Learn to use screen, a terminal multiplexer 2015-12-04T00:00:00+00:00 screen</a>, or GNU screen, is a terminal multiplexer. It allows to manage multiple terminal sessions within the same console. In a way, it does the same thing as modern terminal emulators such as Terminator</a> or Terminology</a> with their built-in tab system and layout management. The main benefit is that screen also works through an SSH connection: you will be able to use your screen knowledge and configuration on any machine supporting screen in the world! screen has been around for almost 30 years so it's pretty stable now. It is released under the GPL license. The main issue I had when starting with screen was its relative difficulty when compared to simple terminal emulators. So I will make a tour of the most useful keyboard shortcuts for a practical use, and share with you my screen config. You can also jump directly to the cheatsheet</a> if you want. Basic usage</h2> screen is not installed by default, so you need to install it first using your package manager. To start screen, open a terminal and run the command screen</code>. Not that hard, is it? We're ready to start learning screen. Don't worry, that's only a few commands! Window management</h3> Now you have opened screen, the first thing you need to know is how to create a new "window" (i.e. a new virtual terminal). To do so, press the keys Ctrl+a</code>, and then press c</code>. The screen shortcuts all consist of Ctrl+a</code> (called "escape key" or "prefix") followed by another key. You now have two windows opened running in parallel, you can have vim in the first one, and a bash in the second one for example, and that's what screen is all about. You can visualize all the opened windows with Ctrl+a</code> "</code>. This view allows you to navigate through your windows with the arrow keys, the 0-9 numbers and the enter key. This command is great because it gives you a global overview of your current session, I used it a lot at first. But there are more ways to navigate through the windows! I personally use almost exclusively the commands Ctrl+a</code> p</code> (previous window) and Ctrl + a</code> n</code> (next window). These shortcuts require to have your windows in mind though, but it comes with practice. And don't forget Ctrl+a</code> "</code> is here to rescue you in case of doubt. ;) It's also possible to jump to another window with the command Ctrl+a</code> number</code>, where number is a number between 0 and 9. This matches the "Num" column of the Ctrl+a</code> "</code> view. Finally, you can kill a window when you have no use for it anymore. To do so, just press Ctrl+d</code> in the shell. There's also a screen command with the "same" effect but it's usually better to proceed this way. Note if you kill all the windows of a session, the session terminates. Summary: Ctrl+a</code> c</code> to create a new window</li> Ctrl+a</code> "</code> to visualize the opened windows</li> Ctrl+a</code> p</code> and Ctrl+a</code> n</code> to switch with the previous/next window</li> Ctrl+a</code> number</code> to switch to the window number</li> Ctrl+d</code> to kill a window</li> </ul> Session management</h3> Now you can manage your windows, it is time you learn how to manage your screen sessions. You just learned you can terminate a session by killing all its windows, that's a good start. We will now see how to exit a session without killing it. It means you'll be able to manage multiple persistent sessions in parallel. This is called "detaching" a session. To be clear, a detached session is a session which is still active in background, but on which no one is connected. An attached session is a session which is active and currently used. The first thing I'll ask you to do is to kill your current screen session if you haven't done it already. You can then list all your screen sessions by running the command screen -ls</code>. If all your sessions are closed, it should print something like: No Sockets found in /var/run/screen/S-thiht.</code></pre> Now, we'll start a new session, but we'll give it a name so that we can come back to it more easily later. Run the command screen -S my_session</code>. If you run the command screen -ls</code>, you should obtain something like: There is a screen on: 10321.my_session (Attached) 1 Socket in /var/run/screen/S-thiht.</code></pre> Since you're connected to the session, it's presented as attached. Now you're connected to a session named "my_session", you can detach it from the current terminal with the command Ctrl+a</code> d</code>. This should result in a message like: [detached from 10321.my_session]</code></pre> As usual, you can list the currently running sessions with screen -ls</code>. This time, "my_session" should be listed as detached: There is a screen on: 10321.my_session (Detached) 1 Socket in /var/run/screen/S-thiht.</code></pre> I suggest you start and detach a few screen sessions to understand how it works. Be careful not to use the same session name twice, screen allows it but it can be more painful to resume a session if there are name conflicts. With multiple screen sessions, the screen -ls</code> gives something like: There are screens on: 10474.downloads (Detached) 10427.work (Detached) 10321.my_session (Detached) 3 Sockets in /var/run/screen/S-thiht.</code></pre> If there aren't any conflicts, you can resume a session simply with the command screen -x session_name</code>. So for example if you run screen -x my_session</code>, your session "my_session" will be resumed with the windows you opened before! And if you run screen -ls</code>, it should appear as attached: There are screens on: 10474.downloads (Detached) 10427.work (Detached) 10321.my_session (Attached) 3 Sockets in /var/run/screen/S-thiht.</code></pre> A last useful thing you should know is how to kill a detached session without connecting to it. To do so, the command to run is a bit tricky: screen -S session_name -X quit</code>. This means you pass the command "quit" to the session named "session_name". It's a bit hard to remember so I generally simply resume my session and kill all its windows one by one. Summary: screen -ls</code> to list the sessions and their status</li> screen -S session_name</code> to start a session with a given name. The name should be unique</li> Ctrl+a</code> d</code> to detach a session</li> screen -x session_name</code> to resume (reattach) a session knowing its name</li> screen -S session_name -X quit</code> to terminate a detached session</li> </ul> Advanced usage</h2> The hardest part is over! You have discovered screen and you know how to use it. All you need now is practice. In this section, I'll present a few interesting features of screen, but they're definitely not necessary to use it, so feel free to come back and read this part later. Layout management</h3> In the introduction, I talked about how Terminator and Terminology can work as layout managers. The good news is screen also permits to do that. So to split horizontally, you can use the command Ctrl+a</code> S</code>, and to split vertically you can use Ctrl+a</code> |</code>. To switch to the next region (this is how the split panes are called), the command to use is Ctrl+a</code> <Tab></code>. Each region behaves normally: you can run any command you have learned so far. To close the current region, use Ctrl+a</code> X</code>. If you work with a lot of panes, you can close them all but the current one with Ctrl+a</code> Q</code>. Summary: Ctrl+a</code> S</code> to split horizontally</li> Ctrl+a</code> |</code> to split vertically</li> Ctrl+a</code> <Tab></code> to switch to the next region</li> Ctrl+a</code> X</code> to close the current region</li> Ctrl+a</code> Q</code> to close all the regions but the current one</li> </ul> Work with the buffer</h3> To follow, I'll talk about an important feature of screen: the copy mode, or scrollback mode. To enter the copy mode, the command is either Ctrl+a</code> <Esc></code> or Ctrl+a</code> [</code>, choose your side. The copy mode allows you to navigate through the console buffer without using your mouse. Since you can't use your wheel to scroll by default, this is the normal way to go back in the results. This mode lets you use some vim commands for the movements (arrow keys, 0</code>, $</code>, g</code>, G</code>, etc.). You can copy some text (that's what the mode is for!) by marking the beginning and the end of a section with the space bar. You can then paste it with Ctrl+a</code> ]</code>. For more informations, check the man screen</code> and look for "copy". Summary: Ctrl+a</code> <Esc></code> or Ctrl+a</code> [</code> to enter in copy mode</li> <Space></code> to select and copy some text in copy mode</li> Ctrl+a</code> ]</code> to paste the content of the buffer</li> </ul> Session sharing</h3> I'll finish with a last screen feature I want you to know about: it allows to share a session between multiple users. When working on a remote server, this comes really handy. So first, you have to start a session as usual. Then, we'll use the command mode of screen to enable the multiuser mode and give the access right to some users. To enter the command mode, type Ctrl+a</code> :</code>. You can then enable the multiuser mode with the command multiuser on. Then to add a specific user to the session, enter the command mode again with Ctrl+a</code> :</code> and type the command acladd username (ACL stands for "Access Control List") with the name of the user you want to add instead of username. The user username is now able to join your session simply with the command screen -x your_username/session_name</code>. The session is completely shared, which means you can see what the other users do in real time and so can they. When a session is shared with multiple users, it can be nice to know who is currently connected on it. You can have this info and more thanks to the command Ctrl+a</code> *</code>. The view shows a few useful informations, namely the window a user is currently using, or their permissions. Finally, you can revoke the access rights of user with the command acldel username in command mode. There's more you can do to manage a shared session, but I won't go into details, the man page covers everything. Now you have a basic understanding of screen, it should be easier to understand. Summary: Ctrl+a</code> :</code> to switch to command mode "multiuser on" to enable the multiuser mode</li> "acladd username" to give the access rights to the user username</li> "acldel username" to remove the access rights to the user username</li> </ul> </li> screen -x owner/session_name</code> to join a session started by another user</li> Ctrl+a</code> *</code> to list the other connections to a session</li> </ul> Practical tips</h2> Automatically list the screen sessions</h3> When I start my terminal, I like it to list the current screen sessions. To do so, I simply added the screen -ls</code> command to my .bashrc. So now whenever I open a terminal, I'm welcomed with my screen status: There are screens on: 2927.my_screen3 (Attached) 2882.my_screen2 (Attached) 2840.my_screen1 (Detached) 3 Sockets in /var/run/screen/S-thiht.</code></pre> Not only is it really useful to have this status without asking for it, but it is also a good way to think about always using screen. Good aliases</h3> As you're now aware, screen uses a lot of options. I tried to keep only the most useful in this article but they're still a pain to remember. A good way to remember them is to use aliases. I personally use the following: alias sn='screen -S' # sn for screen new alias sl='screen -ls' # sl for screen list alias sr='screen -x' # sr for screen resume function sk() { # sk for screen kill # function instead of alias because the order of the parameters matters screen -S "$1" -X quit }</code></pre> These aliases don't conflict with anything standard and they're really easier to remember and to type, so I strongly advise you to use them! Custom configuration</h3> screen can be customized in various ways through a .screenrc file, located in your home folder. One of the most useful tricks is the status bar. I won't go through the configuration of screen in details (you can read man screen</code> or other articles to learn more), but simply share my commented .screenrc with you: # enable the altscreen # for example in vim, it allows to scroll with the mouse without # "overflowing" on the terminal # also it doesn't "pollute" the terminal when you quit vim altscreen on # automatically detach the session if the terminal is closed autodetach on # larger command history defscrollback 10000 # set the encoding of the windows to utf-8 defutf8 on # disable the startup message startup_message off # allow to scroll termcapinfo xterm* ti@:te@ # disable the annoying visual bell vbell off # status bar # %{= wk}: set the color to white on black # %S: screen session name # %-w: previous windows # %{= kw}: set the color to black on white # %n: current window number # %t: current window title # %{= wk}: set the color to white on black # %+w: next windows caption always "%{= wk}%S: %-w%{= kw}%n %t%{= wk}%+w"</code></pre> Note you can enable any of these options directly from screen's command mode. Cheatsheet of the main commands</h3> Basic usage</h4> Command</th> Description</th></tr></thead> screen</code></td> Start a session</td></tr> screen -ls </code></td> List the sessions and their status</td></tr> screen -S session_name</code></td> Start a session named "session_name"</td></tr> screen -x session_name</code></td> Resume the session named "session_name"</td></tr> screen -S session_name -X quit</code></td> Terminate the session named "session_name"</td></tr> </tbody></table> Command</th> Description</th></tr></thead> Ctrl+a</code> c</code></td> Create a new window</td></tr> Ctrl+a</code> k</code></td> Kill the current window (Ctrl+d</code></a> does the same thing)</td></tr> Ctrl+a</code> "</code></td> List the opened windows</td></tr> Ctrl+a</code> p</code>/n</code></td> Go to the previous/next window</td></tr> Ctrl+a</code> 0</code>-9</code></td> Go to the window n</td></tr> Ctrl+a</code> d</code></td> Detach the screen session</td></tr> Ctrl+a</code> :</code></td> Enter the command mode</td></tr> </tbody></table> Split mode</h4> Command</th> Action</th></tr></thead> Ctrl+a</code> S</code></td> Split horizontally</td></tr> Ctrl+a</code> `</td> `</td></tr> Ctrl+a</code> <Tab></code></td> Go to the next region</td></tr> Ctrl+a</code> X</code></td> Close the current region</td></tr> Ctrl+a</code> Q</code></td> Close all the regions but the current one</td></tr> </tbody></table> Copy mode</h4> Command</th> Action</th></tr></thead> Ctrl+a</code> <Esc></code>/[</code></td> Enter the copy mode</td></tr> <Space></code></td> Mark a selection</td></tr> Ctrl+a</code> ]</code></td> Paste the content of the buffer</td></tr> </tbody></table> Shared session</h4> Command</th> Action</th></tr></thead> Ctrl+a</code> :</code> multiuser on</code></td> Enable the multiuser mode</td></tr> Ctrl+a</code> :</code> acladd username</code></td> Give the access rights to username</td></tr> Ctrl+a</code> :</code> acldel username</code></td> Revoke the access rights to username</td></tr> Ctrl+a</code> *</code></td> List the other connections to the session</td></tr> screen -x owner/session_name</code></td> Join a session started by another user</td></tr> </tbody></table> Conclusion</h2> screen is a powerful tool if you use the terminal a lot. Yet, it has some drawbacks, that can be pretty annoying. To list a few you will be confronted to if you really try to use it: bad UTF-8 support. There are a few configurations to make screen work fine with unicode, but it's still a bit messy. I am not even sure there is a proper way to handle UTF-8 characters in the status bar...</li> insane status bar configuration. You can see it in the .screenrc I provided, crafting a neat status bar is really hard. It's a shame given how useful they are!</li> </ul> Good news though, all these issues are allegedly solved with tmux</a>, another more recent terminal mutiplexer. I've only used it as a replacement of screen for a few days so I will wait a bit before reviewing it, but it's very similar to screen in its usage, and I'm very satisfied with it so far.

SQL Transactions in Go: The Good Way

Writing & Testing a Paginated API Iterator in Go

TDD is a personal practice

Shell Scripts Matter

Use Bashisms</h3>
If you use Bash, don't half-use it. Use parameter expansion</a>. Use local</a> and readonly</a> variables. Use improved conditional expressions</a>.</p>

Quote your variables</h3>
Even if you know quoting is not required, it's a good habit to always quote variables. The only exception is when you specifically want expansion to occur. More on word splitting</a>.</p>

Learn to use screen, a terminal multiplexer

Practical tips</h2>

Cheatsheet of the main commands</h3>

Thibaut Rousseau's Blog

The most popular Go dependency is…

Thibaut Rousseau's Blog

The most popular Go dependency is…

SQL Transactions in Go: The Good Way

Writing & Testing a Paginated API Iterator in Go

Writing an Iterator</h2> Iterators can return up to two values. They must implement one of these signatures:</p>

Writing tests for the sequence</h3> To test an iterator, I find it easier to first write down the expected sequence of values it returns. To do so, we can start by defining a type to represent the 2 values returned by an iteration:</p>

TDD is a personal practice

Shell Scripts Matter

Document your scripts</h2> Any shell script should have a --help</code> option. This doesn't seem easy? It is, thanks to the following wizardry:</p>

Use Bashisms</h3> If you use Bash, don't half-use it. Use parameter expansion</a>. Use local</a> and readonly</a> variables. Use improved conditional expressions</a>.</p>

Quote your variables</h3> Even if you know quoting is not required, it's a good habit to always quote variables. The only exception is when you specifically want expansion to occur. More on word splitting</a>.</p>

Use subshells as a way to control what's in your global scope</h3> An example is worth a thousand words:</p>

Stay informed</h3> Shell scripting isn't moving that much these days. It's a great reason to read stuff about the topic, it makes it easy to keep up! Here are some interesting resources:</p>

Learn to use screen, a terminal multiplexer

Practical tips</h2>

Cheatsheet of the main commands</h3>

Use Bashisms</h3>
If you use Bash, don't half-use it. Use parameter expansion</a>. Use local</a> and readonly</a> variables. Use improved conditional expressions</a>.</p>

Quote your variables</h3>
Even if you know quoting is not required, it's a good habit to always quote variables. The only exception is when you specifically want expansion to occur. More on word splitting</a>.</p>