Using ‘Go Generate’ To Deploy Multi-Process Apps

PSP31475-00, 7HA.02 Gas Turbine, Rotor Instalation, Greenville, SC, USA, DI-3800×5700

In microservice architectures, it is fairly common to have a project that includes different worker types. A Makefile can be used to manage the creation of multiple programs, but the Go toolchain has a great tool that can be used as well, go generate. Here are some examples of how we take advantage of ‘go generate’ at Nuvi:

  • API/Worker – We have an API that allows clients to start/stop long-running jobs, and a worker which accesses the same database to run the jobs.
  • NLP Classifier – We have different processes that share a majority of the same code, but have different quirks depending on if they are classifying sentiment, vulgarity, or subjectivity.

In other words, we have one git repository, but from that code, we want to generate ‘n‘ number of executables.

Repository Structure

Our normal single-process repositories have the following structure:

  • main.go
  • foo.go (other main package source files)
  • bar.go
  • internal (folder for packages intended just for this project)
    • database (package for database access)
      • database.go
    • rabbit (package for rabbitmq access)
      • rabbit.go

As you can see, when there is only one program (one main.go) its really easy to build and deploy. We just run:

go build

from the root of the repository.

Structure With Multiple Programs

Now let’s say we have a project that has an API that is responsible for managing some long-running jobs. For example, we can pretend it manages RSS scraping jobs. Here is how we would build out the repository:

  • cmd
    • api
      • main.go
    • worker
      • main.go
  • internal (folder for packages intended just for this project)
    • database (package for database access)
      • database.go
    • rss (package for rss scraping logic)
      • rss.go
  • gen.go

Here we have a cmd folder in the root, which then holds a directory for each executable. This allows us to still scope packages to the entire project, share a CI/CD pipeline, and keep code that is tightly-coupled all in one place.

How Do We Build It?

In the above project structure, you may have noticed the gen.go file. Here is what it looks like:

package main

//go:generate go build ./cmd/api
//go:generate go build ./cmd/worker

Now we can run go generate from the root of our project and both executables will be built at the root.

The generate function is a powerful tool and can do a lot more than just build a list of executables. Nevertheless, we’ve found this to be a convenient way to keep our projects “go native”.

Thanks For Reading

Hit me up on twitter @wagslane if you have any questions or comments.

Follow me on Dev.to: wagslane

The post Using ‘Go Generate’ To Deploy Multi-Process Apps appeared first on Qvault.



source https://qvault.io/2020/04/22/using-go-generate-to-deploy-multi-process-apps/

Comments

Post a Comment

Popular posts from this blog

Why is Exclusive Or (XOR) Important in Cryptography?

Image
If you are getting into cryptography, or just trying to understand the fundamentals, you may have noticed that the exclusive-or operation is used quite often, especially in ciphers. What is XOR ( ⊕ )? XOR, or “exclusive or” operates on binary data. It returns true if both of its inputs are opposites (one false and one true), otherwise, it returns false. An example in go code would be something like: func exclusiveOr(a bool, b bool) bool { return a != b } XOR Cipher – The Perfect Cipher The XOR operation can be used as a simple cipher for encrypting and decrypting messages with a single key. This is known as symmetric encryption. It is interesting to note that if: The key is the same size as the message The key is kept secret and generated truly randomly Then the cipher is impossible to crack. This is known as a one time pad . However, a simple XOR shouldn’t be used in production due to the key length needing to be too long to be practical. Cipher Example A...
Read more

Base64 vs Base58 Encoding

Image
By Lane Wagner –  @wagslane  on Twitter Base64 is one of the most popular encoding formats for representing data. Have some binary data? Base64 encode it for convenient readability and parsing. Base58 is just another encoding format (with 58 characters instead of 64, and has gained popularity largely due to Bitcoin and other cryptocurrencies. When it comes to data encoding, there is typically a trade-off made between: Human Readability: Do humans have a good idea of what is being represented at a glance? Efficient data compression: How many bytes are used to represent the same data? Let’s rank Base58, Base64, and ASCII encoding against each other using these metrics! First: A Note on Binary All data is stored in a raw binary format on computers. These encoding formats (Base64, Base58, and ASCII) are just different ways of reading and writing binary data. For example, in Base64 the binary code 000000 represents the letter A , but in ASCII the binary 00000000 repres...
Read more

(Very) Basic Intro to Hash Functions (SHA-256, MD-5, etc)

Image
This is going to be a basic introduction to hash functions. I will assume most of my audience is here to gain an understanding of why hash functions are used and the basic idea of why they work. My goal is to explain it in a general sense, I will be omitting proofs and implementation details and instead focusing on the high-level principles. Why Use A Hash Function? Hash functions are used all over the internet in order to securely store passwords, find duplicate records, quickly store and retrieve data, and more. For example,  Qvault  uses hashes to extend master passwords into private encryption keys. A longer list of uses here:  https://en.wikipedia.org/wiki/Hash_function#Uses I want to focus on several important features of hash functions, arguably the most important features. Hash functions scramble data deterministically No matter the input, the output of a hash function always has the same size The original data can not be retrieved from the scrambled d...
Read more