The Ultimate Guide to JSON in Go

json data

The post The Ultimate Guide to JSON in Go first appeared on Qvault.

Being a language built for the web, Go offers feature-rich support for working with JSON data. JSON (JavaScript Object Notation) is an unbelievably popular data interchange format whose syntax resembles simple JavaScript objects. It’s one of the most common ways for web applications to communicate.

Encoding and decoding with struct tags

Go takes a unique approach for working with JSON data. The best way to think about JSON data in Go is as an encoded struct. When you encode and decode a struct to JSON, the key of the JSON object will be the name of the struct field unless you give the field an explicit JSON tag.

type User struct {
     FirstName string `json:"first_name"` // key will be "first_name"
     BirthYear int `json:"birth_year"` // key will be "birth_year"
     Email string // key will be "Email"
 }

Example marshal JSON from struct (encode)

The encoding/json package exposes a json.Marshal function that allows us to create the JSON encoding of any type, assuming that type has an encoder implemented. The good news is, all the default types have an encoder, and you’ll usually be working with structs filled with default-type fields.

func Marshal(v interface{}) ([]byte, error)

As you can see, Marshal() takes a value as input, and returns the encoded JSON as a slice of bytes on success, or an error if something went wrong.

dat, _ := json.Marshal(User{
    FirstName: "Lane",
    BirthYear: 1990,
    Email:     "example@gmail.com",
})
fmt.Println(string(dat))

// prints:
// {"first_name":"Lane","birth_year":1990,"Email":"example@gmail.com"}

Example unmarshal JSON to struct (decode)

func Unmarshal(data []byte, v interface{}) error

Similarly, the json.Unmarshal() function takes some encoded JSON data and a pointer to a value where the encoded JSON should be written, and returns an error if something goes wrong.

dat := []byte(`{
    "first_name":"Lane",
    "birth_year":1990,
    "Email":"example@gmail.com"
    }`)
user := User{}
err := json.Unmarshal(dat, &user)
if err != nil {
    fmt.Println(err)
}
fmt.Println(user)
// prints:
// {Lane 1990 example@gmail.com}

Example – Go JSON HTTP server

Building a JSON API in Go is simple, you don’t even need a framework to get access to convenient high-level HTTP support. I typically start by writing two little helper functions, respondWithJSON and responsdWithError.

func respondWithJSON(w http.ResponseWriter, code int, payload interface{}) error {
    response, err := json.Marshal(payload)
    if err != nil {
        return err
    }
    w.Header().Set("Content-Type", "application/json")
    w.Header().Set("Access-Control-Allow-Origin", "*")
    w.WriteHeader(code)
    w.Write(response)
    return nil
}

respondWithJSON makes it easy to send a JSON response by simply providing the handler’s ResponseWriter, an HTTP status code, and a payload to be marshalled (typically a struct).

func respondWithError(w http.ResponseWriter, code int, msg string) error {
    return respondWithJSON(w, code, map[string]string{"error": msg})
}

The respondWithError function wraps the respondWithJSON function and always sends an error message. Now let’s take a look at how to build a full HTTP handler.

func handler(w http.ResponseWriter, r *http.Request) {
    defer r.Body.Close()
    type requestBody struct {
        Email    string `json:"email"`
        Password string `json:"password"`
    }
    type responseBody struct {
        Token string `json:"token"`
    }

    dat, err := ioutil.ReadAll(r.Body)
    if err != nil {
        respondWithError(w, 500, "couldn't read request")
        return
    }
    params := requestBody{}
    err = json.Unmarshal(dat, &params)
    if err != nil {
        respondWithError(w, 500, "couldn't unmarshal parameters")
        return
    }

    // do stuff with username and password

    respondWithJSON(w, 200, responseBody{
        Token: "example-auth-token",
    })
}

Since the json.Marshal and json.Unmarshal function work on the []byte type, it’s really easy to send those bytes over the wire or write them to disk.

Example – Reading and writing JSON files

I use JSON files to store configuration from time to time. Go makes it easy to read and write JSON files.

Write JSON to a file in Go

type car struct {
    Speed int    `json:"speed"`
    Make  string `json:"make"`
}
c := car{
    Speed: 10,
    Make:  "Tesla",
}
dat, err := json.Marshal(c)
if err != nil {
    return err
}
err = ioutil.WriteFile("/tmp/file.json", dat, 0644)
if err != nil {
    return err
}

Read JSON from a file in Go

type car struct {
    Speed int    `json:"speed"`
    Make  string `json:"make"`
}
dat, err := ioutil.ReadFile("/tmp/file.json")
if err != nil {
    return err
}
c := car{}
err = json.Unmarshal(dat, &c)
if err != nil {
    return err
}

Tag Options – Omitempty

When marshalling data you can leave out a key completely if the key’s value contains a zero value using the omitempty tag.

type User struct {
     FirstName string `json:"first_name,omitempty"`"
     BirthYear int `json:"birth_year"`
 }

// if FirstName = "" and BirthYear = 0
// marshaled JSON will be:
// {"birth_year":0}

// if FirstName = "lane" and BirthYear = 0
// marshaled JSON will be:
// {"first_name":"lane","birth_year":0}

Tag Options – Ignore field

As mentioned above, non-exported (lowercase) fields are ignored by the marshaler. If you want to ignore additional fields you can use the - tag.

type User struct {
     // FirstName will never be encoded
     FirstName string `json:"-"`"
     BirthYear int `json:"birth_year"`
 }

Default encoding types

JSON and Go types don’t match up 1-to-1. Below is a table that describes the type relationships when encoding and decoding.

Go Type JSON Type
bool boolean
float64 number
string string
nil pointer null
time.Time RFC 3339 timestamp (string)

You may notice that the float32 and int types are missing. Don’t worry, you absolutely can encode and decode numbers into those types, they just don’t have an explicit type in the JSON spec. For example, if you encode an integer to JSON, you’re guaranteed it won’t have a decimal point. However, if someone mutates that JSON value to be a floating-point number before you decode it you’ll be given a runtime error.

It’s rare to encounter an error when marshaling JSON data, but unmarshaling JSON can frequently cause errors. Here are some things to watch out for:

  • Any type conflicts will result in an error. For example, you can’t unmarshal a string into a int, even if the string value is a stringified number: "speed": "42"
  • A floating-point number can’t be decoded into an integer
  • A null value can’t be decoded into a value that doesn’t have a nil option. For example, if you have a number field that can be null, you should unmarshal into a *int
  • A time.Time can only decode an RFC 3339 string – other kinds of timestamps will fail

Custom JSON marshaling

While most types have a default way to encode and decode JSON data, you may want custom behavior from time to time. Luckily, the json.Marshal and json.Unmarshal respect the json.Marshaler and json.Unmarshaler interfaces. In order to customize your behavior you just need to overwrite their methods MarshalJSON and UnmarshalJSON respectively.

type Marshaler interface {
    MarshalJSON() ([]byte, error)
}
type Unmarshaler interface {
    UnmarshalJSON([]byte) error
}

One of the most common scenarios for me is want to encode and decode timestamps in a different format, usually due to interoperability with another language like JavaScript.

type Group struct {
    ID        string        `json:"id"`
    CreatedAt unixTimestamp `json:"created_at"`
}

type unixTimestamp time.Time

func (ut unixTimestamp) MarshalJSON() ([]byte, error) {
    s := strconv.Itoa(int(time.Time(ut).Unix()))
    return []byte(s), nil
}

func (ut *unixTimestamp) UnmarshalJSON(dat []byte) error {
    unix, err := strconv.Atoi(string(dat))
    if err != nil {
        return err
    }
    *ut = unixTimestamp(time.Unix(int64(unix), 0))
    return nil
}

func main() {
    g := Group{
        ID:        "my-id",
        CreatedAt: unixTimestamp(time.Unix(1619544689, 0)),
    }
    dat, _ := json.Marshal(g)
    fmt.Println(string(dat))
    // prints
    // {"id":"my-id","created_at":1619544689}

    newG := Group{}
    json.Unmarshal(dat, &newG)
    fmt.Println(newG)
    // prints
    // {my-id {0 63755141489 0x1694c0}}
}

Arbitrary JSON with map[string]interface{}

It’s unfortunate when this is the case, but sometimes we have to work with arbitrary JSON data. For example, you need to decode some JSON data, but you aren’t sure what the key structure or shape of the data is.

The best way to handle this case it to unmarshal the data into a map[string]interface{}

dat := []byte(`{
    "first_name": "lane",
    "age": 30
}`)
m := map[string]interface{}{}
json.Unmarshal(dat, &m)
for k, v := range m {
    fmt.Printf("key: %v, value: %v\n", k, v)
}

// prints
// key: first_name, value: lane
// key: age, value: 30

I want to point out that map[string]interface{} should only be used when you absolutely have to. If you have knowledge of the shape of the data, please use a struct or another concrete type. Avoid the dynamic typing provided by interfaces when working with JSON, if you want, you can always use anonymous structs for one-off usage.

Streaming JSON encodings

Sometimes you don’t have the luxury of reading all the JSON data to or from a []byte. If you need to be able to parse data as it’s streamed in or out of your program the encoding/json package provides Decoder and Encoder types.

func NewDecoder(r io.Reader) *Decoder
func NewEncoder(w io.Writer) *Encoder

Take a look at the following example. It decodes data from standard in, adds a new key "id" with a value of "gopher-man" and writes the result to standard out.

dec := json.NewDecoder(os.Stdin)
enc := json.NewEncoder(os.Stdout)
for {
    v := map[string]interface{}{}
    if err := dec.Decode(&v); err != nil {
        log.Fatal(err)
    }
    v["id"] = "gopher-man"
    if err := enc.Encode(&v); err != nil {
        log.Fatal(err)
    }
}

Pretty printing JSON

By default, the json.Marshal function compresses all the whitespace in the encoded data for efficiency. If you need to print out your JSON data so that it’s more easily readable you can pretty print it using the json.MarshalIndent function.

func MarshalIndent(v interface{}, prefix, indent string) ([]byte, error)

You can customize how you want your pretty JSON to be formatted, but if you just want it to be tabbed and newlined correctly you can do the following.

type user struct {
    Name string
    Age  int
}

json, err := json.MarshalIndent(user{Name: "lane", Age: 30}, "", "  ")
if err != nil {
    return err
}

fmt.Println(string(json))
// prints
// {
//     "Name": "lane",
//     "Age": 30
// }

Thanks for reading, now take a course!

Interested in a high-paying job in tech? Land interviews and pass them with flying colors after taking my hands-on coding courses.

Start coding now

Questions?

Follow and hit me up on Twitter @q_vault if you have any questions or comments. If I’ve made a mistake in the article be sure to let me know so I can get it corrected!

Subscribe to my newsletter for more coding articles delivered straight to your inbox.



source https://qvault.io/golang/json-golang/

Comments

Post a Comment

Popular posts from this blog

Why is Exclusive Or (XOR) Important in Cryptography?

Image
If you are getting into cryptography, or just trying to understand the fundamentals, you may have noticed that the exclusive-or operation is used quite often, especially in ciphers. What is XOR ( ⊕ )? XOR, or “exclusive or” operates on binary data. It returns true if both of its inputs are opposites (one false and one true), otherwise, it returns false. An example in go code would be something like: func exclusiveOr(a bool, b bool) bool { return a != b } XOR Cipher – The Perfect Cipher The XOR operation can be used as a simple cipher for encrypting and decrypting messages with a single key. This is known as symmetric encryption. It is interesting to note that if: The key is the same size as the message The key is kept secret and generated truly randomly Then the cipher is impossible to crack. This is known as a one time pad . However, a simple XOR shouldn’t be used in production due to the key length needing to be too long to be practical. Cipher Example A...
Read more

Base64 vs Base58 Encoding

Image
By Lane Wagner –  @wagslane  on Twitter Base64 is one of the most popular encoding formats for representing data. Have some binary data? Base64 encode it for convenient readability and parsing. Base58 is just another encoding format (with 58 characters instead of 64, and has gained popularity largely due to Bitcoin and other cryptocurrencies. When it comes to data encoding, there is typically a trade-off made between: Human Readability: Do humans have a good idea of what is being represented at a glance? Efficient data compression: How many bytes are used to represent the same data? Let’s rank Base58, Base64, and ASCII encoding against each other using these metrics! First: A Note on Binary All data is stored in a raw binary format on computers. These encoding formats (Base64, Base58, and ASCII) are just different ways of reading and writing binary data. For example, in Base64 the binary code 000000 represents the letter A , but in ASCII the binary 00000000 repres...
Read more

(Very) Basic Intro to Hash Functions (SHA-256, MD-5, etc)

Image
This is going to be a basic introduction to hash functions. I will assume most of my audience is here to gain an understanding of why hash functions are used and the basic idea of why they work. My goal is to explain it in a general sense, I will be omitting proofs and implementation details and instead focusing on the high-level principles. Why Use A Hash Function? Hash functions are used all over the internet in order to securely store passwords, find duplicate records, quickly store and retrieve data, and more. For example,  Qvault  uses hashes to extend master passwords into private encryption keys. A longer list of uses here:  https://en.wikipedia.org/wiki/Hash_function#Uses I want to focus on several important features of hash functions, arguably the most important features. Hash functions scramble data deterministically No matter the input, the output of a hash function always has the same size The original data can not be retrieved from the scrambled d...
Read more