Abstract JSON

Abstract JSON is a small golang package provides a parser for JSON with support of JSONPath, in case when you are not sure in its structure.

Method Unmarshal will scan all the byte slice to create a root node of JSON structure, with all its behaviors.

Method Marshal will serialize current Node object to JSON structure.

Each Node has its own type and calculated value, which will be calculated on demand. Calculated value saves in atomic.Value, so it's thread safe.

Method JSONPath will returns slice of found elements in current JSON data, by JSONPath request.

Compare with other solutions

Check the cburgmer/json-path-comparison project.

Usage

Playground

package main

import (
	"fmt"
	"github.com/spyzhov/ajson"
)

func main() {
	json := []byte(`...`)

	root, _ := ajson.Unmarshal(json)
	nodes, _ := root.JSONPath("$..price")
	for _, node := range nodes {
		node.SetNumeric(node.MustNumeric() * 1.25)
		node.Parent().AppendObject("currency", ajson.StringNode("", "EUR"))
	}
	result, _ := ajson.Marshal(root)

	fmt.Printf("%s", result)
}

Console application

You can download ajson cli from the release page, or install from the source:

go get github.com/spyzhov/ajson/cmd/[email protected]

Usage:

Usage: ajson [-mq] "jsonpath" ["input"]
  Read JSON and evaluate it with JSONPath.
Parameters:
  -m, --multiline  Input file/stream will be read as a multiline JSON. Each line should have a full valid JSON.
  -q, --quiet      Do not print errors into the STDERR.
Argument:
  jsonpath         Valid JSONPath or evaluate string (Examples: "$..[?(@.price)]", "$..price", "avg($..price)")
  input            Path to the JSON file. Leave it blank to use STDIN.

Examples:

  ajson "avg($..registered.age)" "https://randomuser.me/api/?results=5000"
  ajson "$.results.*.name" "https://randomuser.me/api/?results=10"
  curl -s "https://randomuser.me/api/?results=10" | ajson "$..coordinates"
  ajson "$" example.json
  echo "3" | ajson "2 * pi * $"
  docker logs image-name -f | ajson -qm 'root($[?(@=="ERROR" && key(@)=="severity")])'

JSONPath

Current package supports JSONPath selection described at http://goessner.net/articles/JsonPath/.

JSONPath expressions always refer to a JSON structure in the same way as XPath expression are used in combination with an XML document. Since a JSON structure is usually anonymous and doesn't necessarily have a "root member object" JSONPath assumes the abstract name $ assigned to the outer level object.

JSONPath expressions can use the dot–notation

$.store.book[0].title

or the bracket–notation

$['store']['book'][0]['title']

for input paths. Internal or output paths will always be converted to the more general bracket–notation.

JSONPath allows the wildcard symbol * for member names and array indices. It borrows the descendant operator .. from E4X and the array slice syntax proposal [start:end:step] from ECMASCRIPT 4.

Expressions of the underlying scripting language (<expr>) can be used as an alternative to explicit names or indices as in

$.store.book[(@.length-1)].title

using the symbol @ for the current object. Filter expressions are supported via the syntax ?(<boolean expr>) as in

$.store.book[?(@.price < 10)].title

Here is a complete overview and a side by side comparison of the JSONPath syntax elements with its XPath counterparts.

Script engine

Predefined constant

Package has several predefined constants.

 e       math.E     float64
 pi      math.Pi    float64
 phi     math.Phi   float64
 
 sqrt2     math.Sqrt2   float64
 sqrte     math.SqrtE   float64
 sqrtpi    math.SqrtPi  float64
 sqrtphi   math.SqrtPhi float64
 
 ln2     math.Ln2    float64
 log2e   math.Log2E  float64
 ln10    math.Ln10   float64
 log10e  math.Log10E float64
      
 true    true       bool
 false   false      bool
 null    nil        interface{}
 

You are free to add new one with function AddConstant:

    AddConstant("c", NumericNode("speed of light in vacuum", 299_792_458))

Examples

Playground

package main

import (
	"fmt"
	
	"github.com/spyzhov/ajson"
)

func main() {
	json := []byte(`{"foo": [true, null, false, 1, "bar", true, 1e3], "bar": [true, "baz", false]}`)
	result, _ := ajson.JSONPath(json, `$..[?(@ == true)]`)
	fmt.Printf("Count of `true` values: %d", len(result))
}

Output:

Count of `true` values: 3

Playground

package main

import (
	"fmt"
	
	"github.com/spyzhov/ajson"
)

func main() {
	json := []byte(`{"foo": [true, null, false, 1, "bar", true, 1e3], "bar": [true, "baz", false]}`)
	result, _ := ajson.JSONPath(json, `$..[?(@ == true)]`)
	fmt.Printf("Count of `true` values: %d", len(result))
}

Output:

Count of `true` values: 3

Supported operations

Package has several predefined operators.

Operator precedence

Precedence    Operator
    6	    	  **
    5             *  /  %  <<  >>  &  &^
    4             +  -  |  ^
    3             ==  !=  <  <=  >  >= =~
    2             &&
    1             ||

Arithmetic operators

**   power                  integers, floats
+    sum                    integers, floats, strings
-    difference             integers, floats
*    product                integers, floats
/    quotient               integers, floats
%    remainder              integers

&    bitwise AND            integers
|    bitwise OR             integers
^    bitwise XOR            integers
&^   bit clear (AND NOT)    integers

<<   left shift             integer << unsigned integer
>>   right shift            integer >> unsigned integer

==  equals                  any
!=  not equals              any
<   less                    any
<=  less or equals          any
>   larger                  any
>=  larger or equals        any
=~  equals regex string     strings

You are free to add new one with function AddOperation:

	AddOperation("<>", 3, false, func(left *ajson.Node, right *ajson.Node) (node *ajson.Node, err error) {
		result, err := left.Eq(right)
		if err != nil {
			return nil, err
		}
		return BoolNode("neq", !result), nil
	})

Examples

Playground

package main

import (
	"fmt"
	
	"github.com/spyzhov/ajson"
)

func main() {
	json := []byte(`[{"name":"Foo","mail":"[email protected]"},{"name":"bar","mail":"[email protected]"}]`)
	result, err := ajson.JSONPath(json, `$.[?(@.mail =~ '.+@example\\.com')]`)
	if err != nil {
		panic(err)
	}
	fmt.Printf("JSON: %s", result[0].Source())
	// Output:
	// JSON: {"name":"Foo","mail":"[email protected]"}
}

Output:

JSON: {"name":"Foo","mail":"[email protected]"}

Supported functions

Package has several predefined functions.

abs          math.Abs           integers, floats
acos         math.Acos          integers, floats
acosh        math.Acosh         integers, floats
asin         math.Asin          integers, floats
asinh        math.Asinh         integers, floats
atan         math.Atan          integers, floats
atanh        math.Atanh         integers, floats
avg          Average            array of integers or floats
b64decode    b64 Decoding       string
b64encode    b64 Encoding       string 
b64encoden   b64 Encoding (no padding)    string
cbrt         math.Cbrt          integers, floats
ceil         math.Ceil          integers, floats
cos          math.Cos           integers, floats
cosh         math.Cosh          integers, floats
erf          math.Erf           integers, floats
erfc         math.Erfc          integers, floats
erfcinv      math.Erfcinv       integers, floats
erfinv       math.Erfinv        integers, floats
exp          math.Exp           integers, floats
exp2         math.Exp2          integers, floats
expm1        math.Expm1         integers, floats
factorial    N!                 unsigned integer
first        Get first element  any
floor        math.Floor         integers, floats
gamma        math.Gamma         integers, floats
is_array     Is type Array      any
is_bool      Is type Bool       any
is_float     Is type Float      any
is_int       Is type Int        any
is_null      Is type Null       any
is_numeric   Is type Numeric    any
is_object    Is type Object     any
is_string    Is type String     any
is_uint      Is type Uint       any
j0           math.J0            integers, floats
j1           math.J1            integers, floats
key          Key of element     string
last         Get last element   any
length       Length of array    array, string
log          math.Log           integers, floats
log10        math.Log10         integers, floats
log1p        math.Log1p         integers, floats
log2         math.Log2          integers, floats
logb         math.Logb          integers, floats
not          not                any
parent       Get parent element any
pow10        math.Pow10         integer
rand         N*rand.Float64     float
randint      rand.Intn          integer
root         Get root element   any
round        math.Round         integers, floats
roundtoeven  math.RoundToEven   integers, floats
sin          math.Sin           integers, floats
sinh         math.Sinh          integers, floats
size         Count of elements  array, object
sum          Sum                array of integers or floats
sqrt         math.Sqrt          integers, floats
tan          math.Tan           integers, floats
tanh         math.Tanh          integers, floats
trunc        math.Trunc         integers, floats
y0           math.Y0            integers, floats
y1           math.Y1            integers, floats

You are free to add new one with function AddFunction:

	AddFunction("trim", func(node *ajson.Node) (result *Node, err error) {
		if node.IsString() {
			return StringNode("trim", strings.TrimSpace(node.MustString())), nil
		}
		return
	})

Examples

Playground

package main

import (
	"fmt"
	
	"github.com/spyzhov/ajson"
)

func main() {
	json := []byte(`{"prices": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}`)
	root, err := ajson.Unmarshal(json)
	if err != nil {
		panic(err)
	}
	result, err := ajson.Eval(root, `avg($.prices)`)
	if err != nil {
		panic(err)
	}
	fmt.Printf("Avg price: %0.1f", result.MustNumeric())
	// Output:
	// Avg price: 5.5
}

Output:

Avg price: 5.5

Examples

Calculating AVG(price) when object is heterogeneous.

{
  "store": {
    "book": [
      {
        "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      {
        "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      },
      {
        "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      },
      {
        "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    },
    "tools": null
  }
}

Unmarshal

Playground

package main

import (
	"fmt"
	"github.com/spyzhov/ajson"
)

func main() {
	data := []byte(`{"store": {"book": [
{"category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95}, 
{"category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99}, 
{"category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99}, 
{"category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99}], 
"bicycle": {"color": "red", "price": 19.95}, "tools": null}}`)

	root, err := ajson.Unmarshal(data)
	if err != nil {
		panic(err)
	}

	store := root.MustKey("store").MustObject()

	var prices float64
	size := 0
	for _, objects := range store {
		if objects.IsArray() && objects.Size() > 0 {
			size += objects.Size()
			for _, object := range objects.MustArray() {
				prices += object.MustKey("price").MustNumeric()
			}
		} else if objects.IsObject() && objects.HasKey("price") {
			size++
			prices += objects.MustKey("price").MustNumeric()
		}
	}

	if size > 0 {
		fmt.Println("AVG price:", prices/float64(size))
	} else {
		fmt.Println("AVG price:", 0)
	}
}

JSONPath:

Playground

package main

import (
	"fmt"
	"github.com/spyzhov/ajson"
)

func main() {
	data := []byte(`{"store": {"book": [
{"category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95}, 
{"category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99}, 
{"category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99}, 
{"category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99}], 
"bicycle": {"color": "red", "price": 19.95}, "tools": null}}`)

	nodes, err := ajson.JSONPath(data, "$..price")
	if err != nil {
		panic(err)
	}

	var prices float64
	size := len(nodes)
	for _, node := range nodes {
		prices += node.MustNumeric()
	}

	if size > 0 {
		fmt.Println("AVG price:", prices/float64(size))
	} else {
		fmt.Println("AVG price:", 0)
	}
}

Eval

Playground

package main

import (
	"fmt"
	"github.com/spyzhov/ajson"
)

func main() {
	json := []byte(`{"store": {"book": [
{"category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95}, 
{"category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99}, 
{"category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99}, 
{"category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99}], 
"bicycle": {"color": "red", "price": 19.95}, "tools": null}}`)
	root, err := ajson.Unmarshal(json)
	if err != nil {
		panic(err)
	}
	result, err := ajson.Eval(root, "avg($..price)")
	if err != nil {
		panic(err)
	}
	fmt.Println("AVG price:", result.MustNumeric())
}

Marshal

Playground

package main

import (
	"fmt"
	"github.com/spyzhov/ajson"
)

func main() {
	json := []byte(`{"store": {"book": [
{"category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95}, 
{"category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99}, 
{"category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99}, 
{"category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99}], 
"bicycle": {"color": "red", "price": 19.95}, "tools": null}}`)
	root := ajson.Must(ajson.Unmarshal(json))
	result := ajson.Must(ajson.Eval(root, "avg($..price)"))
	err := root.AppendObject("price(avg)", result)
	if err != nil {
		panic(err)
	}
	marshalled, err := ajson.Marshal(root)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s", marshalled)
}

Benchmarks

Current package is comparable with encoding/json package.

Test data:

{ "store": {
    "book": [ 
      { "category": "reference",
        "author": "Nigel Rees",
        "title": "Sayings of the Century",
        "price": 8.95
      },
      { "category": "fiction",
        "author": "Evelyn Waugh",
        "title": "Sword of Honour",
        "price": 12.99
      },
      { "category": "fiction",
        "author": "Herman Melville",
        "title": "Moby Dick",
        "isbn": "0-553-21311-3",
        "price": 8.99
      },
      { "category": "fiction",
        "author": "J. R. R. Tolkien",
        "title": "The Lord of the Rings",
        "isbn": "0-395-19395-8",
        "price": 22.99
      }
    ],
    "bicycle": {
      "color": "red",
      "price": 19.95
    }
  }
}

JSONPath: $.store..price

$ go test -bench=. -cpu=1 -benchmem
goos: linux
goarch: amd64
pkg: github.com/spyzhov/ajson
BenchmarkUnmarshal_AJSON          138032              8762 ns/op            5344 B/op         95 allocs/op
BenchmarkUnmarshal_JSON           117423             10502 ns/op             968 B/op         31 allocs/op
BenchmarkJSONPath_all_prices       80908             14394 ns/op            7128 B/op        153 allocs/op

License

MIT licensed. See the LICENSE file for details.