YottaDB Go Wrapper

Towards our goal of “YottaDB Everywhere”, we have invested heavily in exposing YottaDB’s core data management. In r1.20, we released a highly performant C Simple API with minimal software between the API and the data management engine. In r1.24, we extended that API to support multi-threaded applications. As the lingua franca of computing, accessible from all programming languages, a C API allows us to create “wrappers” to provide application developers in popular languages with native APIs, each of which aims to fit well with the culture and paradigms of its language.

The first language for which we have created a wrapper is the popular Go language.

As recounted by Go co-designer and developer Rob Pike:

… Go’s design considerations include rigorous dependency management, the adaptability of software architecture as systems grow, and robustness across the boundaries between components. … Go is a compiled, concurrent, garbage-collected, statically typed language developed at Google. It is an open source project: Google imports the public repository rather than the other way around. .… Go is efficient, scalable, and productive.

While those are good reasons to create a Go wrapper for YottaDB, the language has features that threw down a gauntlet of technical challenges that we had to overcome.

Instead of YottaDB’s traditional model of concurrency based on processes, global variables, locks and optimistic-concurrency control, Go’s abstractions are at a higher level, implementing the abstractions of Communicating Sequential Processes using independently executing coroutines (called Goroutines) dynamically scheduled over a set of operating system threads within a process. This means that the YottaDB runtime system cannot use a thread-id to identify requests, and must instead use a context, called a “tptoken”, as described in our blog post YottaDB Support for Multi-threaded Applications.

Go blends the features of procedural, as well as object oriented, languages. While Go has functions, like those of procedural languages, a Go structure can have associated methods. A method is conceptually like a function whose first parameter is a pointer to a structure. In designing the API, we had to make decisions as to whether to package each unit of functionality as a function or a method. For the purpose of this blog post, “function” is used generically to mean either a function or a method, whereas “method” specifically refers to methods operating on structures.

As one of the design goals of Go is safety, pointers to Go’s heap space and to functions or methods are not permitted to be passed to non-Go code such as the YottaDB runtime, Since data from YottaDB is often passed back to YottaDB – for example, getting the next subscript at a level in the tree and accessing that subtree, or getting data from a cross-reference node and using it as a subscript – we felt that minimizing copying by cgo of data between Go and YottaDB heaps was important for efficiency and performance. This led to a design decision to leave the data in YottaDB, and to pass pointers to that data between Go and YottaDB, allowing Go application code to manipulate pointers rather than data, using the pointers to access or provide data when needed. However, while such a design may be computationally efficient, it has (at least) two consequences:

1. Memory management is conceptually harder than allowing strings to be copied back and forth on each call.
2. YottaDB cannot know when a structure allocated in its heap is out of scope for Go application code, and should be freed or garbage collected, in order to prevent memory leaks.

Our solution to the first was to create two families of functions, an “Easy API” and a “Simple API”. The Easy API is a set of functions (not methods) that copy strings between Go and YottaDB on each call. The solution to the latter was to use finalizers to free YottaDB structures when the Go structures pointing to them go out of scope.

Our goal with wrappers for YottaDB is for each wrapper to fit in as much as possible with the ecosystem and culture of its language. This means that on a system or container with YottaDB installed, installing and using the Go wrapper is as simple as:

• Install with:

  go get lang.yottadb.com/go/yottadb

• To use, include

  import “lang.yottadb.com/go/yottadb”

  in application code

• Documentation is at https://godoc.org/lang.yottadb.com/go/yottadb (with complete documentation in the Multi-Language Programmers Guide)

We are also providing a Dockerfile including the packaged Go wrapper.

Please use the YottaDB Go wrapper, and tell us what you think!

References:

• IEEE Spectrum’s Interactive: The Top Programming Languages 2018
• The RedMonk Programming Language Rankings: June 2018
• TIOBE Programming Community Index

Images Used:
Go Gopher, Renee French.
The design is licensed under the Creative Commons 3.0 Attributions license.
Read this article for more details: https://blog.golang.org/gopher

Published on February 06, 2019

See All Announcements