Pitfalls of GoLang interface streaming to JSON (part3)

Photo by Deon Black on Unsplash

In the first two sections, I walked through a method to allow the reconstruction of interface types from JSON streams. To accomplish this, I added a new field to the parent message to indicate the type of the struct in the stream.

type StructX struct {
   X           string      `json:"x"`
   MyInterface MyInterface `json:"my_interface"`
}

in the stream becomes

var xr struct {
   X               string      `json:"x"`
   MyInterface     MyInterface `json:"my_interface"`
   MyInterfaceType Type        `json:"my_interface_type"`
}

Please read part1 and part2 to see where this construction comes from.

Pitfalls of GoLang interface streaming to JSON (part1)

Pitfalls of GoLang interface streaming to JSON (part2)

What happens when we get presented with an embedded type in a JSON stream? I’ll use the same types created in part1 for consistency.

[ 
   {"$type":"StructA", "a":4.56},
   {"$type":"StructA", "a":1.23},
   {"$type":"StructB", "b":"lazy"}
]

Here we have an array of heterogeneous types, all of which implement our MyInterface interface.

Naively we would expect the following to work but on reflection, we know that it will not and will throw an error.

func main() {
   b := []byte(`[ 
      {"$type":"StructA", "a":4.56},
      {"$type":"StructA", "a":1.23},
      {"$type":"StructB", "b":"lazy"}
   ]`)
   data := make([]MyInterface, 0)

   err := json.Unmarshal(b, &data)
   if err != nil {
      panic(err)
   }
   fmt.Printf("%s", data)
}

It’s the same error we had before, Go does not know how to unmarshal into an interface type.

panic: json: cannot unmarshal object into Go value of type main.MyInterface

Ok, so we can bind a custom unmarshal method to the array of MyInterface, not so fast.

func (l []MyInterface) UnmarshalJSON( b []byte) error {
   return nil 
}

If you try and compile you will get the following error

.\main.go:104:7: invalid receiver type []MyInterface

If you dig a little deeper this is because []MyInterface is an unnamed type and Go does not allow receivers to be bound. This is an easy fix,

type MyList []MyInterface

func (l *MyList) UnmarshalJSON(b []byte) error {
   return nil
}

We can name the type and Go will happily allow us to add methods. Note as mentioned in part1 the receiver of the Unmarshal function should be a pointer. This allows us to modify the type and if we don’t do this, we could not allocate the length of the slice.

This type naming is one of the most powerful parts of the Go-type system and can be used in many cases in place of complex logic. In part1 we used a string alias to allow us to have type safety on our streaming type. We can also bind additional methods to that type, for example, implementing the stringer interface to provide a custom output.

Now when we call json.Unmarshal we have to supply our named type, this ensures that we will call our custom method.

var data MyList
err := json.Unmarshal(b, &data)

Inside our unmarshal method we now have a byte array that represents the entire list of JSON objects. We could start to do some string math and peek inside not recommended or we could break this up into a slice of bytes. If you remember json.RawMessage, we used this to perform a lazy decode previously, we can do the same here.

func (l *MyList) UnmarshalJSON(b []byte) error {
   var raw []json.RawMessage
   err := json.Unmarshal(b, &raw)
   if err != nil {
      return err
   } 
   return nil
}

We didn’t have to validate the JSON, nor did we have to write a parser to understand where the end of each element existed. There is a downside here though, Go is applying its parser to the JSON, though I’m not convinced I would do better and likely much worse. So, what do we do with this data now that we have it? Well, we learned that we could use json.Unmarshal against a custom type and if that type just has one field of $type then we can extract the type of the object from the list.

type LazyType struct {
   Type Type `json:"$type"`
}

We are going to use a named type here, but we could equally have defined this inline. Now we can loop across the list of binaries.

// Allocate an array of MyInterface
*l = make(MyList, len(raw))
var t LazyType
for i, r := range raw {
   // Unmarshal the array first into a type array
   err := json.Unmarshal(r, &t)
   if err != nil {
      return err
   }
}

It’s worth looking at the first line after the comment where we allocated the size of the receiver to be the same length as our list of bytes. Without a pointer receiver, we could not accomplish this. Finally, we add in the logic we used in Part2 to create the instances of our interface. I’ll add the whole decoder here.

func (l *MyList) UnmarshalJSON(b []byte) error {
   var raw []json.RawMessage
   err := json.Unmarshal(b, &raw)
   if err != nil {
      return err
   }
   // Allocate an array of MyInterface
   *l = make(MyList, len(raw))
   var t LazyType
   for i, r := range raw {
      // Unmarshal the array first into a type array
      err := json.Unmarshal(r, &t)
      if err != nil {
         return err
      }
      // Create an instance of the type
      myInterfaceFunc, ok := lookup[t.Type]
      if !ok {
         return fmt.Errorf("unregistered interface type : %s", t.Type)
      }
      myInterface := myInterfaceFunc.New()
      err = json.Unmarshal(r, myInterface)
      if err != nil {
         return err
      }
      (*l)[i] = myInterface
   }
   return nil
}

and there we have it; we can take our JSON and convert it to our heterogeneous list of structs. If I run the whole, I get the expected output.

[%!s(*main.StructA=&{4.56}) %!s(*main.StructA=&{1.23}) %!s(*main.StructB=&{lazy})]

A slice of two StructA’s and one StructB with the expected payloads. Personally, I find it satisfying to lean into the built-in functionality in this way vs start to hand roll my own JSON parsers. The core downside is that we do parse and iterate across the JSON list three times to get to our result.

Let me know what you think about this solution, and I’d love to hear from people who have other ways to accomplish these streaming tasks.

Full runnable sample

package main

import (
   "encoding/json"
   "fmt"
)

type Type string
type MyInterface interface {
   Type() Type
   New() MyInterface
}

var lookup = make(map[Type]MyInterface)

func Register(iface MyInterface) {
   lookup[iface.Type()] = iface
}

func init() {
   Register(StructA{})
   Register(StructB{})
}

type StructA struct {
   A float64 `json:"a"`
}
type StructB struct {
   B string `json:"b"`
}


func (_ StructA) Type() Type {
   return "StructA"
}

func (_ StructB) Type() Type {
   return "StructB"
}

func (_ StructA) New() MyInterface {
   return &StructA{}
}

func (_ StructB) New() MyInterface {
   return &StructB{}
}

// Check that we have implemented the interface
var _ MyInterface = (*StructA)(nil)
var _ MyInterface = (*StructB)(nil)


type LazyType struct {
   Type Type `json:"$type"`
}

type MyList []MyInterface

func (l *MyList) UnmarshalJSON(b []byte) error {
   var raw []json.RawMessage
   err := json.Unmarshal(b, &raw)
   if err != nil {
      return err
   }
   // Allocate an array of MyInterface
   *l = make(MyList, len(raw))
   var t LazyType
   for i, r := range raw {
      // Unmarshal the array first into a type array
      err := json.Unmarshal(r, &t)
      if err != nil {
         return err
      }
      // Create an instance of the type
      myInterfaceFunc, ok := lookup[t.Type]
      if !ok {
         return fmt.Errorf("unregistered interface type : %s", t.Type)
      }
      myInterface := myInterfaceFunc.New()
      err = json.Unmarshal(r, myInterface)
      if err != nil {
         return err
      }
      (*l)[i] = myInterface
   }
   return nil
}

func main() {
   b := []byte(`[ 
      {"$type":"StructA", "a":4.56},
      {"$type":"StructA", "a":1.23},
      {"$type":"StructB", "b":"lazy"}
   ]`)

   var data MyList
   err := json.Unmarshal(b, &data)
   if err != nil {
      panic(err)
   }
   fmt.Printf("%s", data)
}