ProductPromotion
Logo

Go.Lang

made by https://0x3d.site

GitHub - seiflotfy/skizze: A probabilistic data structure service and storage
A probabilistic data structure service and storage - seiflotfy/skizze
Visit Site

GitHub - seiflotfy/skizze: A probabilistic data structure service and storage

GitHub - seiflotfy/skizze: A probabilistic data structure service and storage

Build Status license

Skizze ([ˈskɪt͡sə]: german for sketch) is a sketch data store to deal with all problems around counting and sketching using probabilistic data-structures.

Unlike a Key-Value store, Skizze does not store values, but rather appends values to defined sketches, allowing one to solve frequency and cardinality queries in near O(1) time, with minimal memory footprint.

Current status ==> Alpha (tagged v0.0.2)

Motivation

Statistical analysis and mining of huge multi-terabyte data sets is a common task nowadays, especially in areas like web analytics and Internet advertising. Analysis of such large data sets often requires powerful distributed data stores like Hadoop and heavy data processing with techniques like MapReduce. This approach often leads to heavyweight high-latency analytical processes and poor applicability to realtime use cases. On the other hand, when one is interested only in simple additive metrics like total page views or average price of conversion, it is obvious that raw data can be efficiently summarized, for example, on a daily basis or using simple in-stream counters. Computation of more advanced metrics like a number of unique visitor or most frequent items is more challenging and requires a lot of resources if implemented straightforwardly.

Skizze is a (fire and forget) service that provides a probabilistic data structures (sketches) storage that allows estimation of these and many other metrics, with a trade off in precision of the estimations for the memory consumption. These data structures can be used both as temporary data accumulators in query processing procedures and, perhaps more important, as a compact – sometimes astonishingly compact – replacement of raw data in stream-based computing.

Example use cases (queries)

  • How many distinct elements are in the data set (i.e. what is the cardinality of the data set)?
  • What are the most frequent elements (the terms “heavy hitters” and “top-k elements” are also used)?
  • What are the frequencies of the most frequent elements?
  • How many elements belong to the specified range (range query, in SQL it looks like SELECT count(v) WHERE v >= c1 AND v < c2)?
  • Does the data set contain a particular element (membership query)?

How to build and run

make dist
./bin/skizze

Bindings

Two bindings are currently available:

Example usage:

Skizze comes with a CLI to help test and explore the server. It can be run via

./bin/skizze-cli

Commands

Create a new Domain (Collection of Sketches):

#CREATE DOM $name $estCardinality $topk
CREATE DOM demostream 10000000 100

Add values to the domain:

#ADD DOM $name $value1, $value2 ....
ADD DOM demostream zod joker grod zod zod grod

Get the cardinality of the domain:

# GET CARD $name
GET CARD demostream

# returns:
# Cardinality: 9

Get the rankings of the domain:

# GET RANK $name
GET RANK demostream

# returns:
# Rank: 1	  Value: zod	  Hits: 3
# Rank: 2	  Value: grod	  Hits: 2
# Rank: 3	  Value: joker	  Hits: 1

Get the frequencies of values in the domain:

# GET FREQ $name $value1 $value2 ...
GET FREQ demostream zod joker batman grod

# returns
# Value: zod	  Hits: 3
# Value: joker	  Hits: 1
# Value: batman	  Hits: 0
# Value: grod	  Hits: 2

Get the membership of values in the domain:

# GET MEMB $name $value1 $value2 ...
GET MEMB demostream zod joker batman grod

# returns
# Value: zod	  Member: true
# Value: joker	  Member: true
# Value: batman	  Member: false
# Value: grod	  Member: true

List all available sketches (created by domains):

LIST

# returns
# Name: demostream  Type: CARD
# Name: demostream  Type: FREQ
# Name: demostream  Type: MEMB
# Name: demostream  Type: RANK

Create a new sketch of type $type (CARD, MEMB, FREQ or RANK):

# CREATE CARD $name
CREATE CARD demosketch

Add values to the sketch of type $type (CARD, MEMB, FREQ or RANK):

#ADD $type $name $value1, $value2 ....
ADD CARD demostream zod joker grod zod zod grod

License

Skizze is available under the Apache License, Version 2.0.

Authors

Articles
to learn more about the golang concepts.

Resources
which are currently available to browse on.

mail [email protected] to add your project or resources here 🔥.

FAQ's
to know more about the topic.

mail [email protected] to add your project or resources here 🔥.

Queries
or most google FAQ's about GoLang.

mail [email protected] to add more queries here 🔍.

More Sites
to check out once you're finished browsing here.

0x3d
https://www.0x3d.site/
0x3d is designed for aggregating information.
NodeJS
https://nodejs.0x3d.site/
NodeJS Online Directory
Cross Platform
https://cross-platform.0x3d.site/
Cross Platform Online Directory
Open Source
https://open-source.0x3d.site/
Open Source Online Directory
Analytics
https://analytics.0x3d.site/
Analytics Online Directory
JavaScript
https://javascript.0x3d.site/
JavaScript Online Directory
GoLang
https://golang.0x3d.site/
GoLang Online Directory
Python
https://python.0x3d.site/
Python Online Directory
Swift
https://swift.0x3d.site/
Swift Online Directory
Rust
https://rust.0x3d.site/
Rust Online Directory
Scala
https://scala.0x3d.site/
Scala Online Directory
Ruby
https://ruby.0x3d.site/
Ruby Online Directory
Clojure
https://clojure.0x3d.site/
Clojure Online Directory
Elixir
https://elixir.0x3d.site/
Elixir Online Directory
Elm
https://elm.0x3d.site/
Elm Online Directory
Lua
https://lua.0x3d.site/
Lua Online Directory
C Programming
https://c-programming.0x3d.site/
C Programming Online Directory
C++ Programming
https://cpp-programming.0x3d.site/
C++ Programming Online Directory
R Programming
https://r-programming.0x3d.site/
R Programming Online Directory
Perl
https://perl.0x3d.site/
Perl Online Directory
Java
https://java.0x3d.site/
Java Online Directory
Kotlin
https://kotlin.0x3d.site/
Kotlin Online Directory
PHP
https://php.0x3d.site/
PHP Online Directory
React JS
https://react.0x3d.site/
React JS Online Directory
Angular
https://angular.0x3d.site/
Angular JS Online Directory