While YottaDB r1.34 is an otherwise modest successor to r1.32, internal changes allow the popular programming language Python to be fully Supported. We are excited about making YottaDB available to the large Python user community. Here is a “Hello, World” Python program that updates the database and illustrates YottaDB’s Unicode support:
import yottadb if __name__ == "__main__": yottadb.set("^hello", ("Python",), value="नमस्ते दुनिया")
The Python wrapper can be installed with pip install yottadb. Full details of the API are in the Python wrapper user documentation. The current Debian Docker image at Docker Hub includes the Python wrapper. We thank Peter Goss (@gossrock) for his contributions to the Python wrapper.
Python joins C, Go, M, node.js, Perl, and Rust as languages with APIs to access YottaDB.
Owing to an internal change required to support the Python wrapper, application code written in Go and Rust will need to be compiled with new versions of the Go and Rust wrappers. We anticipate no regressions, and apologize for the inconvenience.
As discussed in our blog post Fuzz Testing YottaDB, adding a new type of testing exposed bugs previously neither encountered in our development environment nor reported by a user. Although fuzz testing generates syntactically correct but semantically questionable, contorted code that is unlikely to be part of any real application, the bugs are nevertheless defects. r1.34 includes the first tranche of fixes. As we are dedicating hardware to continuous fuzz testing, future YottaDB releases will include fixes for bugs found by fuzzing. We thank Zachary Minneker of Security Innovation for Fuzz Testing YottaDB and bringing its benefits to our attention.
In addition to fixes for issues, whether found by fuzz testing or otherwise, YottaDB r1.34 has enhancements that make it faster and more friendly, e.g.,
– Faster stringpool garbage collection, thanks to Alexander Sergeev and Konstantin Aristov (@littlecat).
– HOME and END keys work in direct mode for READ, thanks to Sergey Kamenev (@inetstar).
– Multiple improvements to ydbinstall / ydbinstall.sh.
– Enhancements to ydb_env_set to improve performance under some conditions and to be compatible with existing environments created without ydb_env_set.
– Enhancements to the %RSEL utility program.
YottaDB r1.34 also inherits enhancements and fixes from GT.M V6.3-011.
Details are in the release notes.YottaDB r1.32 Released
Although there is no single theme to YottaDB r1.32, it qualifies as a major release because it includes a significant number of enhancements, including several to enable the Application Independent Metadata plugin. The plugin provides functionality for applications to push responsibility for maintaining cross references and statistics to YottaDB triggers, thereby reducing the code that applications must maintain. Enhancements include:
- The $ZYSUFFIX() function enables applications to create variable and routine names that are guaranteed for all practical purposes to be unique, as it uses the 128-bit MurmurHash3 non-cryptographic hash.
- A $ZPARSE() option to follow symbolic links.
- Sourcing ydb_env_set creates and manages a three-region database, and defaults to UTF-8 mode.
- An option to allow $ZINTERRUPT to be invoked for the USR2 signal.
- The --aim option of ydbinstall / ydbinstall.sh installs the Application Independent Metadata plugin.
- Shell-like word expansion with $VIEW(“WORDEXP”).
- Propagation downstream of context set by triggers.
- ydb_ci_*() functions return the ZHALT argument for C code that calls M code which terminates with a ZHALT.
There are numerous smaller enhancements that make system administration and operations (DevOps) friendlier, and easier to automate. For example, %PEEKBYNAME() has an option to query global directory segments without opening the corresponding database files, orphaned relinkctl files are automatically cleaned up, and the --octo option of ydbinstall / ydbinstall.sh installs Octo such that octo --version reports the git commit hash of the build.2020 Holiday Greetings
We wish you and yours a Merry Christmas, Happy Hannukah, Joyous Kwanzaa, Bright Karthigai – whatever you choose to celebrate this holiday season – and a Happy New Year. We hope all is well with you and yours, and that 2021 brings more sanity than 2020 did.
YottaDB moved to 100% telecommuting in mid-March, and we have been working remotely since then. We anticipate returning to the office after the pandemic; meanwhile, the office is for computers and the occasional cobweb.
Nevertheless, we have been busy this year. Our major milestone was the release of Octo 1.0, for querying YottaDB databases using SQL-92 and JDBC. Our next major target for Octo is read-write capability using SQL. Although the following is very much code that is under development, this demonstration of INSERT INTO will give you a taste of what we plan to release in 2021:
$ echo "create table usstates(state varchar primary key, capital varchar);" | octo OCTO> create table usstates(state varchar primary key, capital varchar); OCTO> $ echo "create table uspresidents (id integer primary key, firstname varchar, middlename varchar, lastname varchar)" | octo OCTO> create table uspresidents (id integer primary key, firstname varchar, middlename varchar, lastname varchar) OCTO> $ head -3 ~/.yottadb/usstates.sql INSERT into USSTATES values ("Alabama", "Montgomery"), ("Alaska", "Juneau"), $ head -3 ~/.yottadb/uspresidents.sql INSERT into USPRESIDENTS values (1, "George", "", "Washington"), (2, "John", "", "Adams"), $ octo -f ~/.yottadb/usstates.sql $ octo -f ~/.yottadb/uspresidents.sql $ echo "select state, capital, id, firstname, middlename, lastname from uspresidents, usstates where lastname = state or lastname = capital;" | octo OCTO> select state, capital, id, firstname, middlename, lastname from uspresidents, usstates where lastname = state or lastname = capital; Washington|Olympia|1|George||Washington Wisconsin|Madison|4|James||Madison Mississippi|Jackson|7|Andrew||Jackson Nebraska|Lincoln|16|Abraham||Lincoln OCTO> $
Along with a number of other enhancements and fixes, in our mission to make YottaDB a language-agnostic database, we added support for Rust this year, adding to our support for C, Go and M. From our community, we have support for access to the database from Perl (thanks, Stefan Traby!), and wrappers for access from node.js (thanks, David Wicksell!) and Python (thanks, Peter Goss!) are being worked on by community members. We continue to merge upstream code releases into the YottaDB code base, while ensuring that the merged code meets the standards for robustness, performance, and security that you have come to expect from us.
Since all our software is developed as a 100% free / open source project, you can follow our work at https://gitlab.com/YottaDB. You can see what our development plans are, and you can even see our review comments on our developers’ code Merge Requests. Please do feel free to add your comments to any Issue or Merge Request.
YottaDB continues to grow. We have grown organically from the original three founders to a team of ten. In 2020, we had a table at an international event, FOSDEM 2020, in Brussels. We had more visitors there than all our events in the US in 2019 combined, but after that the pandemic shut down our in-person events. We hope to resume them in due course of time. In 2020, we also refreshed our web site, and we had additional users who permitted us to feature them on our web site.
We send you our hopes and best wishes for good health, happiness and every success in 2021, and hope that we have an opportunity to meet in person in the not too distant future.What is an Acculturation Guide?
Visiting family in Copenhagen over the holidays at the end of 2019, we were surprised to find Danes carrying packages of fireworks in local trains, which we in the United States would not dream of. Starting around 8pm on New Years Eve, there was a steady and increasing popping of fireworks, and starbursts overhead, leading to a continuous crescendo around midnight. There were fireworks in streets with pedestrian and vehicular traffic, and at one point orange volcano fireworks on the sidewalk across the street were so bright and fiery that they had us rushing to the window to investigate. Crowds gather at Rådhuspladsen, a big plaza outside the Town Hall, with the air thick with smoke. That is Danish culture. Other cultures may celebrate their new year by wearing traditional clothes.
So what is culture? According to Wikipedia, it “encompasses the social behavior and norms found in human societies, as well as the knowledge, beliefs, arts, laws, customs, capabilities and habits of the individuals in these groups.” Coming from cultures where you decline until you are pressed when your dinner host or hostess asks whether you want seconds, many an international guest at an American dinner has been dismayed to see dishes carried back to the kitchen!
Acculturation is a process in which an individual adopts, acquires and adjusts to a new cultural environment. As humans, we learn by doing, and immersion is the only road to acculturation.
What does Acculturation mean for software such as YottaDB? You can of course start with “Hello, world” examples that access the database, but what then? The function of a database is persistence: to store data entrusted to it, and provide the stored data when requested. You need to know how to install the software, configure databases to provide business continuity when systems fail or when an entire data center fails, and more.
To that end, we have created the YottaDB Acculturation Guide. It sits between the “Hello, world“ examples and the YottaDB Administration and Operations Guide, which is complete reference material that can appear overwhelming after “Hello, world”.
The one constant about culture is change! Culture is not static. The original Acculturation Guide (sometimes referred to as the Acculturation Workshop) is over ten years old, but has been significantly revamped to reflect YottaDB’s evolution as a language-agnostic NoSQL database, and simpler crash-recovery. As YottaDB gains new functionality and new operational capabilities, we will edit the Acculturation Guide from time to time, adding new material and updating existing content.
 The Gumpa dance is a special dance celebrated around the time of Losar, the Tibetan New Year. Pictured here in Lachung monastery, North Sikkim, Sikkim, India. Photo by Nichalp downloaded from Wikimedia Commons.
As YottaDBⓇ reaches the two and a half year mark, the year-end season is an opportunity for us to reflect on where we started, and where our journey has brought us.
In 2017 three of us started YottaDB, having each worked on GT.M for over twenty years. With a sense of accomplishment, we could point to our contributions. But the GT.M database would never be directly accessed by any language other than M and we believed that the database had value to applications beyond those written in M (see The Anchovy Pizza Company – Not!).
At the end of 2019, we have organically grown to a team of ten. Our office in Malvern, which felt empty when we moved in, will feel cramped when we add our next couple of team members. We anticipate moving to a bigger office when our current lease ends.
In a word, the goal of YottaDBⓇ is persistence. Any application, in any language, and on any platform, should be able to entrust its data to YottaDB. While that may be an overambitious dream, we have a roadmap.
We started by creating a native C API to the database engine. We did this in two stages, first a set of functions for single-threaded applications, and then additional functions for multi-threaded applications. As C is the lingua franca of computer languages because all programming languages have the ability to call C code and be called by it, a high-performance, native C API opened the door to YottaDB access from other languages, using “wrappers”. Our goal with each wrapper is to provide a robust, high-performance API that a programmer in that language would consider intuitive. The first language for which we created a wrapper is Go. Programming YottaDB in Go is now fully Supported with production grade code. In 2020, we hope to release wrappers for Rust and Python, with more languages to follow.
As all of our work is 100% free / open source software (FOSS), our community also contributes software to the YottaDB ecosystem. For example:
- Nodem (thanks, David Wicksell!) provides a Node.js API to YottaDB.
- A Perl pod (thanks, Stefan Traby!) provides access from Perl.
- CFMumps (thanks, John Willis!) provides Cold Fusion Markup Language access to YottaDB.
- The M Web Server (thanks, Sam Habiel and Christopher Edwards!) provides web services from YottaDB.
- Chris Munt’s GitHub page (thanks, Chris Munt!) provides numerous interface packages for YottaDB, as well as his own native SQL engine.
Although YottaDB is a hierarchical key-value NoSQL database, able to handle schemas that are a superset of relational schemas, many applications have relational schemas, and even applications with non-relational schemas have relational subsets of their schemas. To accommodate the large number of tools for analytics and reporting of relational data, we are investing heavily developing OctoⓇ, a SQL engine to access relational data in YottaDB databases. Octo CREATE statements define tables and their mapping to YottaDB global variables. Octo compiles SQL queries to native object code that directly accesses YottaDB global variables with minimal overhead.
Throughout our work, we have maintained upward compatibility with upstream GT.M releases. We integrate the code of GT.M releases into the YottaDB code base. As we do this, we create automated regression tests, find, and fix bugs. Upgrading from GT.M to YottaDB is functionally 100% upward compatible, and follows the same steps as upgrading from an older GT.M version to a newer version (conversely, for M application applications, downgrading from YottaDB to GT.M follows the same steps as downgrading from a newer GT.M version to an older one).
YottaDB is a fully open project. This means that when we fix an issue or add an enhancement, under our GitLab projects you can see the code changes, you can see review comments and responses, and you can see associated automated regression tests. So when our release notes mention a fix or enhancement to the software, you have complete traceability to the associated code changes and tests. This traceability adds significant value beyond just having the code for a release available under a FOSS license.
In closing, we would like to take this opportunity to wish you a Merry Christmas, Happy Hanukah, Joyous Kwanzaa – whatever you celebrate this season – as well as a Wonderful New Year in 2020. You make YottaDB possible. We thank you and look forward to serving you in 2020.
The YottaDB Team
in alphabetic order: Bhaskar, Brad, Chris, Ganesh, Jon, Joshua, Matt, Narayanan, Saraswathi, and Steve.
with wags from: Dash and Emmy
For users of YottaDB using the Go API, r1.28 is a major release because it is required for production grade access to YottaDB from Go. For others, it is a minor release with a small set of enhancements and fixes as detailed in the complete release notes.
As discussed in our blog post YottaDB Go Wrapper on February 6, 2019, we have created a package yottadB to access YottaDB from the Go language, which is among the top 10 in the IEEE Spectrum 2019 language popularity survey. Where we previously considered the Go wrapper to be field test grade, we now consider it to be production grade.
The reason that it takes a YottaDB release to make the Go wrapper production grade arises from the fact that runtime system of the most commonly used Go implementation assumes that it is in primary control of the process. As the YottaDB runtime system resides within the address space of the application process, it must protect its memory from Go. Furthermore, it must install its own signal handlers. Overcoming these hurdles, which in some cases resulted in intermittent failures that caused abnormal process termination sometimes after hours and even days of stress testing on our servers, was in equal measures art and engineering, which prolonged the release date. Our presentation Hidden Dragons of CGO discusses some of these challenges.
On a system or container with YottaDB installed, installing and using the Go wrapper is as simple as:
- Install with:
go get lang.yottadb.com/go/yottadb
- To use, include
in application code
- Documentation is at https://godoc.org/lang.yottadb.com/go/yottadb (with complete documentation in the Multi-Language Programmers Guide)
We are also providing a Dockerfile including the packaged Go wrapper.
Please use the YottaDB Go wrapper, and tell us what you think!