Author Archives: Jaahanavee Sikri

Comparing YottaDB Web Framework Performance

We welcome Ram Sailopal back for his second guest blog post. Read his first post on the YottaDB Dashboard here.

Background and Motivation

With its “many languages, one database” philosophy, the daemonless, language-agnostic database YottaDB, provides in-process APIs from multiple languages — C, C++, Go, Lua, M, Node.js, Perl, Python, and Rust today. Some language APIs (“wrappers”) are developed by YottaDB, and some by others in its ecosystem. While every language has its philes and phobes, there are entirely rational reasons to select a language, including:

  • Developers & Tool Chains – If you have developers skilled in a language or an available proven tool chain in a language, it maybe simpler to use that language than to train your developers in another language, or to implement a new tool chain.
  • Performance & Throughput – Users need to get data quickly, and as the number of users ramps up, any performance degradation should be gradual.
  • Robustness – Since it is anathema to deliver incorrect results, all language stacks return correct results. However some stacks may experience drop-outs, lost connections, and timeouts, which detract from the user experience.

The basic model of a web stack consists of a web server and a back-end.

It is interesting to compare the performance of different web stacks and frameworks under simulated stress. To compare apples-to-apples, the database, the JSON string response to a REST query, and front-end load generator were the same.

Of course, this end-to-end test only involves a single operation. Any real application consists of many operations at different layers in the framework, only a fraction of which are database accesses.

Test Setup

The database consisted of the following hierarchical key-value tuples:

^PATIENTS(1,"address")="234,Timbucktwo Road, Fantasy City" 
^PATIENTS(1,"age")=52 
^PATIENTS(1,"name")="Bob Taylor"
^PATIENTS(1,"sex")="Male"

which was delivered by the back-end in response to a REST query as the following JSON string:

[{ "id": "1", "name": "Bob Taylor", "age: "52", "sex": "Male", "address: "234,Timbucktwo Road, Fantasy City" }]

Each test stack used two containers both hosted on Gitpod. The front-end load was generated by Locust, running in a container with the web server and the back-end running in another.

Note: In some cases, where the same software is capable of being both the web server and the back-end, separate copies were run, in order to keep the comparison apples-to-apples. In specific situations, such as an in-house application, it may of course be appropriate to run a single copy of the software as both the web server and the back-end.

Apache is used as the web server, including for mgweb-server, which incorporates Apache. The table below lists the back-end that was used for each language. Each test container has a name for identification, and the names link to the detailed results for that stack. Native C and the Perl wrapper were not tested.

Language Test Container Web Server Back-end
C++ yottac Apache Crow+Informatik Aalen CPP-Access-for-YottaDB
Go yottago Apache Gin+YottaDB Go Wrapper
Lua yottalua Apache Pegasus+YottaDB Lua Wrapper
M yottamgweb mgweb-server
M yottaweb Apache M-Web-Server
Node.js nodem Apache Express+Nodem
Node.js mgdbx Apache mg-dbx
Python yottapython Apache Flask + YottaDB Python Wrapper
Python yottamg_python Apache mg_python
Rust yottarust Apache Rocket+YottaDB Rust Wrapper

 

In each case, the load simulated 1,000 concurrent users, and each test involved approximately 100,000 requests.

As expected, running a stress test against a multi-layer application running in one container generated errors, characterized as follows:

  • A web server starts servicing a request, runs out of resources, and is unable to complete the request.
  • A web server lacking the resources to accept a resource for servicing.
  • A crash while servicing a request.

The detailed results for each stack include the errors for that stack.

Results

Click here — Overall Results — to view the overall results, summarized below.

The surprising result is that the winners are the Lua and Rust frameworks, both delivering better performance as well as fewer errors than the native M frameworks. The C++ stack is interesting, because while it delivers good performance, its failures make it an outlier.

All tests are available at YottaWeb-Test. Please do try the tests in your environment, and if you have enhancements to add or results to submit, please submit a Pull Request on GitHub.

About Raman Sailopal

I am a professional with over 20 years of experience working in the IT industry. I reside in Birmingham, UK. My very first corporate interaction with IT involved using M technology within the UK medical sector and although my journey with IT has seen me follow paths involving Linux, Cloud architecture and open source, I have always held a fond regard for M based technologies and their potential with critical IT systems. Working with YottaDB allows me to combine these passions and contribute to the growth of both M and open source.

 

 

 

Images Used:
[1] World wide web. Photo by Svilen.milev.
[2] BCL – spiderweb. Photo by vastateparksstaff.
Screenshot courtesy Raman Sailopal.

Fuzz Testing YottaDB

Summary

Every day, we find fault with our software, so that you don’t!

Robustness in software is a mark of quality that’s often easy to lose in development.

Thanks to Zachary Minneker of Security Innovation, Inc., we are implementing fuzz testing to make our software even more robust. Fuzz testing provides us with one more way to generate test cases to test that the software does not do what it is not supposed to do. As expected from a new form of testing, we have discovered bugs that we did not know existed, and which no user has reported to us.

Fuzzing

The word “fuzz” is hundreds of years old. More recently, it has been used in music for deliberate distortion, typically guitar sounds. Fuzzing as a test method is relatively recent. At its most basic, fuzzing involves running a piece of software over and over with generated inputs, usually invalid, while attempting to exercise as much code as possible. As fuzzing has developed and the state of the art has changed, fuzzing techniques have grown more elaborate and now involve complex tooling and techniques.

Rejection of invalid input by the software, by discarding it, rejecting it, or reporting it, are all appropriate behaviors, depending on the software specification. Behavior such as process termination (unless that is the specified behavior for invalid input), inconsistent internal state (such as overwriting memory), or worse yet, undesirable behavior are bugs to be fixed.

Fuzzing YottaDB

Zachary Minneker from Security Innovation tested YottaDB by feeding the compiler manipulated M code. While it is certainly possible to input randomly generated bytes, most of the input would be rejected by the parser, and the fuzzing would not be efficient, even though it would theoretically eventually expose every issue with the software (see the Infinite Monkey Theorem). To fuzz efficiently, Zachary started with a corpus of M source code from the YottaDB automated test suite, and made some minor changes to YottaDB to make unexpected behavior more easily detected. Then a 24-core machine running the prepared fuzzing harness and corpus fuzzed YottaDB for a number of days, over the course of several months.

He used AFL++ to generate fuzzed input to YottaDB. AFL++ works by using special tracing instrumentation which marks conditional paths through YottaDB. During execution, the exact path taken through the code is found by noting which conditional paths are taken using a mutated input. If an input causes new paths to be taken it is added to the corpus for further fuzzing, which then increases the amount of code paths taken.

Since YottaDB has signal handlers for normal operation, as well as to catch signals in order to shut down cleanly, he commented out the signals as they would interfere with the testing. In some cases, he used AddressSanitizer, a tool which makes otherwise difficult to detect memory bugs crash the application.

YottaDB can respond to input M code in different ways:

  • Flag a syntax error. Since YottaDB is expected to generate syntax errors for syntactically incorrect code, this input is not very interesting from a testing point of view.
  • Run the code. This is also not very interesting from a fuzz testing point of view: even if the code adds 2 and 2 to get 5, that should be caught by normal functional testing.
  • A YottaDB built with the address sanitizer reports an address error. This is YottaDB failure to be captured.
  • Crash. This is potentially a YottaDB failure, because except in cases such as a process sending itself a signal with $ZSIGPROC(), processes should not terminate abnormally.
  • Hang. A hang by itself does not tell the fuzz tester much because the fuzzed code might include a syntactically correct infinite loop. But a hang might indicate a memory bug.

Some bugs may have multiple ways in which they can be triggered. Zachary eliminated duplicates, created minimal inputs for each failure, and performed root cause analysis. After months of work, he identified failures that are captured as CVEs and as a YottaDB Issue against which our development team is creating fixes.

Practical Implications

Failures identified by Zachary’s fuzz testing are potential security issues because:

  • A crash is in principle a denial of service.
  • An error that overflows a buffer, or overwrites memory is in principle a vector for attack if an attacker is able to load and execute an exploit into memory that is overwritten.
  • An error that causes bad object code to be generated is in principle a vector for attack if an attacker can load code to be executed in to the object code.

As a practical matter:

  • The failures are caused by code which is syntactically correct, but functionally useless in the context of an application. Any YottaDB deployment is controlled code that has gone through developers and is tested: your YottaDB application certainly would not execute code from the web without validating it.
  • Exploiting any of the vulnerabilities requires the ability to modify the M code which your system is executing, which requires login access to the system.
  • Except for process crashes, where the crash itself is the exploit, vulnerabilities are chinks in the armor, not exploits. Turning these vulnerabilities into an exploit (which requires M code to be executed) would require an order of magnitude, or more, of effort.

Controlling access to your system, and validating M code executed, are two key layers of defenses against all vulnerabilities.

What Next?

Since security is a journey, not a destination:

  • We are accelerating the YottaDB r1.34 release with fixes identified by Zachary’s fuzzing, and expect to release it in mid Q1 2022, rather than later in the year. This means that some other issues targeted for r1.34 will be deferred to r1.36.
  • Unlike tests in our automated test suites, where you can run a test and have it pass or fail, fuzz testing is a type of random walk that you run continuously till you find something interesting. We will dedicate hardware to continuous fuzzing. As it finds issues, we will fix them in releases of YottaDB.
  • We will start fuzz testing Octo.

In Conclusion

Although perfection does not exist in this universe, you can be assured of our continued commitment to your YottaDB experience being as rock solid, lightning fast, and secure as we can make it.

In closing, we would like to reiterate our thanks to Zachary Minneker and Security Innovation for introducing us to fuzz testing.

 

 

 

 

Images Used:
[1] Fuzz – Rock en Seine. Photo by Alexandre Pennetier.
[2] Keeley Fuzz Bender pedal. Photo by Skimel.
Screenshots courtesy Zachary Minneker.

2021 Holiday Greetings

It is hard to believe that another year has gone by. As always, we wish you and yours a Merry Christmas, Happy Hannukah, Joyous Kwanzaa, Bright Karthigai and a Happy New Year – plus any festive occasion for the season that we may have missed.

A year ago, we hoped that by now we would have put the pandemic behind us, and that life would go back to being “normal”. In 2021, we came to the conclusion that we need to adapt to a new normal. Some of us moved away and became permanent telecommuters. Some of us are mostly working from the office, being more productive there. Some of us have a hybrid model, working partly at home and partly at the office. One of us (Sam) is doing something unique: he has been staying at motels and rentals near national parks, using the opportunity to explore the great outdoors. He recently completed a rim-to-rim hike of the Grand Canyon, which the rest of us can only experience vicariously! The picture above is from that hike.

We are well on the way to an Octo release with read-write capability. INSERT, UPDATE, and DELETE commands are completed, and we have SQL and operational functionality remaining that we think should be part of that milestone. While we hoped to release this in 2021, we would prefer not to fall short of your expectations with respect to the robustness of the software, and our current target is Q1 2022.

You can always download and use what we have at any time. As discussed in our blog post YottaDB Continuous Integration / Continuous Delivery, thanks to our CI pipeline, the master branches of our source code repositories have the latest code that passes all tests, and is production grade. The exception is the core YottaDB software, releases of which involve multiple cycles of an extensive test suite across a network of machines, testing which cannot yet be made into a CI pipeline. The ydbinstall.sh script has options to download and install YottaDB as well as plugins. Each language wrapper is installed the way that developers of that language expect packages to be installed (e.g., the Go wrapper is installed the way that Go developers expect Go packages to be installed).

For 2022, we wish you good software, good documentation, good health, happiness, and much more. We wish for 2022 to be your best year yet. Please stay in touch!

A Puzzling Performance Problem

We thank Yazeed Smadi for his guest post on the YottaDB blog and hope there are many more to follow. If you would like to post on the YottaDB blog, please contact us at info@yottadb.com.

Background

At one of the biggest VistA EMR sites, Hakeem (VistA adapted to the requirements of healthcare in Jordan) was running well, with one exception: when running a MUPIP REORG operation for the database defragmention, the system had an IO bottleneck. After tracking the issue and linking it to the database growth and usage trends, the root cause was found to be related to low Hard Disk Drive (HDD) performance which could not perform a MUPIP REORG operation without impacting user experience.

To resolve this issue, we decided to upgrade the server’s HDD to Solid State Disks (SSDs). After installing the new disks the performance became unpredictable and the application performance degraded the user experience .

Symptoms

Following were the symptoms we observed, along with degraded application performance :

  • The new SSDs were operating as expected, with no IO delays or bottlenecks.
  • There were no processes in the “D” state, waiting for IO.
  • Apart from degraded performance, the application was behaving normally and delivering correct results.
  • The only anomaly was that the CPU run queue increased in a semi-linear trend, sometimes reaching 120, but the server had 86 CPUs running at 100% capacity.

Solution

It took us about fifteen hours of troubleshooting, to find a solution: Set a tuned profile to throughput-performance profile:

tuned-adm profile throughput-performance
systemctl restart tuned

During the troubleshooting phase; it was clear that the CPU queue was going high while it shouldn’t have! This was causing the poor system performance. The only fact that brought to mind the idea of enhancing CPU performance by setting the tuned profile to throughput-performance , thus forcing the CPUs to work at maximum performance, was https://github.com/redhat-performance/tuned/blob/master/profiles/throughput-performance/tuned.conf.

After enabling this profile, the system performance improved, and the CPU queue became empty in a few seconds!

The Root Cause

After successfully resolving the issue, based on IT operation management strategy in EHS, we initiated the problem management process to find the root cause and to prevent it from happening in the future.

The server model used for this site is HP ProLiant DL380 Gen9, which has a BIOS setting for CPC (Collaborative Power Control) set to OS Control mode (located under Power Management -> Power Profile -> Advanced Power Options -> Collaborative Power Control). This feature enables the OS kernel to use Processor Clocking Control (PCC). When CPC is set to OS Control mode; the OS will control the use of intel_pstate, and since the initial OS tuned profile was set to ondemand; the OS did not utilize the full CPU performance and under production workload it lead to low performance.

The OS installed on the new disks is RedHat Enterprise Linux 7.9 (RHEL); which installs the CPU driver pcc-cpufreq driver for this server model. Using this driver will set the tuned profile to ondemand by default.

Therefore; we recommend overcoming this issue by using one of the following solutions:

  1. Set the tuned profile to throughput-performance: this profile will set the maximum performance for the server’s CPUs, because it includes the following CPU settings
               [cpu]
               governor=performance
               energy_perf_bias=performance
               min_perf_pct=100
  2. Disable CPC in the server’s BIOS by changing it to Disabled

Lesson

When doing system performance enhancement by upgrading hardware, it is mandatory to review software performance related settings!

About Yazeed Smadi

Yazeed Smadi leads the Linux System Administration team at Electronic Health Solutions, Amman Jordan. A graduate of German Jordanian University and Technische Hochschule Deggendorf, he worked on blockchain software at T-Mobile in Germany before returning to Jordan. Yazeed is passionate about DevOps automation and has scripted automated YottaDB switchovers to take place in under one minute using Jenkins. Since the world’s best falafel is found in Jordan, Yazeed and Bhaskar together try to find the best falafel in Amman when Bhaskar visits EHS!

YottaDB Continuous Integration / Continuous Delivery

At YottaDB, we always want to get you our latest software (moving from the “release early, release often” mantra of free/open source software (FOSS) to “release continuously”) without compromising our foundation of “Rock Solid. Lightning Fast. Secure.”

With our Continuous Integration / Continuous Delivery (commonly abbreviated together as CI/CD) pipeline, the latest YottaDB software is ready for you to run at any time – you do not have to wait for a formal release. It is as simple as:

  • git clone the repository
  • Follow the steps in the README to build and install the software

All YottaDB software, except the YottaDB database, has a GitLab CI/CD pipeline of functional tests for code changes to be merged. The YottaDB database goes beyond this (see below).

Confidence in software quality is a prerequisite for CI/CD, and software quality is what builds that confidence.

Software Quality

Although there are many ways to measure software quality, all of them boil down to confidence in the software. This means having confidence not just that the software does what it is supposed to do, but also that the software does not do what it is not supposed to do.

While it is hard enough to test that software does what it is supposed to do, how does one test that software is not doing what it is not supposed to do? While it is not practical to test that software does nothing that it is not supposed to do, a good proxy is to test that it continues to do everything that it is supposed to do, even for small changes.

Testing for everything that software is supposed to do requires repeatability, which in turn requires automated testing.

Our YottaDB development process includes automated testing. Code changes have corresponding automated tests that distinguish between the “before” and “after” behaviors of the change. Both the code and automated tests go through peer reviews, and changes in response to review comments are approved by the reviewer. In those rare cases where automated tests are impractical (handlers for unusual error conditions, minor performance improvements, etc.), the code changes go through more critical peer reviews.

It does not suffice for us to have confidence in our software. You, our users, must have confidence in our software. To that end, code changes we make are approved by reviewers, and visible under our projects at GitLab. Comments made in the review process are public, as well as responses, and the actual merge.

While the process is not perfect – software bugs will not be eliminated in our lifetimes, if ever – it does engender confidence in the software, which is the best we can do, given that perfection does not exist in our daily lives.

CI/CD Pipeline

With automated tests, we create GitLab CI pipelines. A pipeline consists of a set of tests that collectively either pass or fail. When we create GitLab Merge Request pipelines for our packages with rules that require the pipeline for a package to pass before a Merge Request is merged into the master branch of that package, we gain confidence not only that the Merge Request does what it is supposed to do, but also that it does not do what it is not supposed to do.

Installing YottaDB Software

Prerequisites

Binary distributions are provided only for the YottaDB database.

Our other supported software consists of :

  • wrappers that provide APIs to YottaDB from languages such as Go, Perl, Python, and Rust (the YottaDB database includes access from both C and M), and
  • plugins such as Octo for SQL access are installed by cloning the appropriate repository, and following instructions to build and install the software.

This means you must have the build tool chain for each software installed on the system. The prerequisites are listed in the README for each software package.

Should a production system not have tool chains installed, e.g., for security reasons, you can build and install the software in another machine, in a virtual machine, or in a container in the same directory as the production system, and then clone the directory with the installed software to the production machine ensuring that ownership and permissions are maintained.

While the installation instructions for wrappers depend on the requirements of the language for which a wrapper provides an API, the YottaDB database and plugins can be installed using the ydbinstall.sh script.

The ydbinstall.sh Script

The ydbinstall.sh script (which when packaged with a YottaDB binary distribution is named ydbinstall), downloads and installs YottaDB and plugins:

  • Copy or download the script from https://gitlab.com/YottaDB/DB/YDB/-/blob/master/sr_unix/ydbinstall.sh into a temporary directory.
  • Make it executable.
  • Run it. For example, to download and install the latest YottaDB binary distribution, and to download, build, and install the Octo plugin, you can execute sudo ./ydbinstall.sh –utf8 default –octo – as the script manages dependencies, specifying –octo also downloads and installs the POSIX plugin, which Octo needs.
  • The –from-source option allows you to git clone the YottaDB database repository, build it, and install it, like any other YottaDB software. This allows you to get the latest YottaDB software without waiting for a binary distribution.
  • ./ydbinstall.sh –help gives you a full list of options. If YottaDB is already installed, you can use it to install, add, or update plugins.

YottaDB Binary Distributions

A large part of the code in the YottaDB database provides error recovery, replication, large databases, etc., features that are not practicable to test in a CI pipeline. This is analogous to the fact that you cannot test the seat belt of a car by driving it, no matter how far you drive it. Yet, if you are going to drive that car every day for years, you must have confidence that the seat belt will be there to protect you in the unlikely event of a crash.

At YottaDB we have a network of a couple dozen machines that include all Supported platforms, ranging from Raspberry Pi Zeroes to x86_64 servers. We run the YottaDB test system across these machines most weekends, and frequently overnight as well. The normal run checks out the current master source code, builds it, and runs it on these machines against the current master test system. The source code is built and tested two ways, “DBG” builds with asserts that detect error and out-of-design conditions for us to debug, and “PRO” builds that are built the way binary distributions are delivered, and which invoke the recovery code for these error and out-of-design conditions. The tests involve extensive simulated crashes and recovery. They also randomly vary a number of parameters to test the various permutations and combinations of operating conditions.

Our released binary distributions are PRO builds that have passed the test system on our machines, and therefore are the ones we recommend for production use.

Since code changes must pass the test system, the ydbinstall.sh script also offers options to clone, build, and install the current master code branch at GitLab without awaiting a formal binary release. These builds are suitable for use in non-production development and testing environments.

The continuous testing of our software means that once a feature or a fix has made it into the master branch of the source code, we can provide a supported customer with a production grade distribution with that feature or fix in about a week.

Our CI/CD process allows you to have your cake and eat it too.

 

Pipe Theme Images Used:

[1] Console of the Wannamaker organ at Philadelphia. Photo by col_adamson downloaded from Wikimedia Commons.

[2] Astronaut and Expedition 64 Flight Engineer Shannon Walker of NASA connects new Gigabit Ethernet cables linking the Unity, Tranquility, and U.S. Destiny laboratory modules aboard the International Space Station. Photo by NASA Video and Image Library downloaded from Wikimedia Commons.

[3] A bagpiper and member of the Queen’s Bands (Queen’s University, Kingston, Ontario) wearing a full plaid in traditional highland dress. Photo by Greenmind downloaded from Wikimedia Commons.

[4] Centre Georges Pompidou, Paris, France. Photo by RG72 downloaded from Wikimedia Commons.

YottaDB Dashboard

Introduction

Users of YottaDB know the benefits of using it. Its technology was established decades ago and it is used in critical medical and financial systems worldwide. Performance overall is “lightning fast” but performance is not simply a binary consideration. A system may perform well at one point in time with a certain workload and not as well at another. As an IT professional, be it a SysAdmin in a traditional IT role or a DevOps engineer in a more modern one, it is important to continuously capture and monitor performance. Monitoring performance allows for the establishing of baselines, including daily, weekly, and monthly cycles, to create trendlines, not only for capacity planning, but also for detecting, diagnosing, and remedying issues. Since human beings interpret visuals well, tools like Prometheus and Grafana make it easier to visualize captured performance data.

Performance Considerations

The causes of performance degradation may be at a system level in terms of CPU or memory that may need scaling but equally it may be at a software/database level in terms of changes in code that may have been released. With modern day DevOps practices adopted along with Continuous Deployment and sometimes multiple releases a day, the monitoring of the effects of software releases is critical, and indeed imperative, to any DevOps process.

Monitoring performance is useful not only for capacity planning and dealing with problems. It can also add business value by identifying times when there is greater demand on a system. Identifying times of greater demand can allow for a better coordination of “targeted” marketing campaigns. For example, if monitoring were to detect a spike in activity from college towns on the first Tuesday each month, that might lead to an insight that perhaps, that is a day when scholarships are disbursed, and students are buying more beers.

YottaDB Performance metrics

At present, there are monitoring capabilities built into YottaDB. The options are as follows:

Additionally there is Gvstat:

All three options provide valuable information such as global sets, gets and kills but the information is somewhat “one dimensional” when viewed and more suited to consumption by software than by humans, e.g:

BTD:124,CAT:221,CFE:0,CFS:0,CFT:0,CFavg:0,CFsigma:0,CFvar:,CQS:0,CQT:0,CQavg:0,CQsigma:0,CQvar:,CTN:199,CYS:0,CYT:0,CYavg:0,CYsigma:0,CYvar:,DEX:0,DFL:2,DFS:1,DRD:17,DTA:60,DWT:124,GET:345,JBB:23192,JEX:0,JFB:286720,JFL:2,JFS:7,JFW:70,JRE:1,JRI:0,JRL:198,JRO:4,JRP:5,KIL:1,LKF:0,LKS:2,LKfrate:0,NBR:1627,NBW:159,NR0:0,NR1:0,NR2:0,NR3:0,NTR:555,NTW:199,ORD:144,QRY:0,SET:197,TBR:0,TBW:0,TC0:0,TC1:0,TC2:0,TC3:0,TC4:0,TR0:0,TR1:0,TR2:0,TR3:0,TR4:0,TRB:0,TTR:0,TTW:0,ZPR:1,ZTR:0

Further details about these metrics can be found here in the YottaDB documentation.

Enhancing Usability

In order to translate this information into a format that better allows trend analysis, we need to be able to graph the data against time. This is where YottaDB-dashboard can help.

YottaDB-dashboard utilizes the following technology stack:

  • Grafana – An open source interactive metrics visualization tool.
  • Prometheus – An open source metrics querying tool that integrates with Grafana.
  • YDBGvstat (referenced earlier)

Utilizing YDBGvstat, YottaDB-dashboard runs an “exporter” presenting the YottaDB metrics periodically via a web server process. Taking an example YottaDB SET metric, the web server process will present data in the example form:

# HELP SET_pit Number of SET operations (TP and non-TP)
# TYPE SET_pit gauge
SET_pit{Process="Yottadb_Stats"} 197.0

This data is then “consumed” by Prometheus, interpreted, and then presented to Grafana via a Prometheus plugin.

Setting Up

  1. Install Grafana – https://grafana.com/docs/grafana/latest/installation/ – or from your Linux distribution’s package manager, if available.
  2. Install Prometheus – https://prometheus.io/docs/prometheus/latest/installation/ – or, like Grafana, from your Linux distribution’s package manager.
  3. Run the YottaDB metrics “exporter” by following the instructions at https://github.com/RamSailopal/YottaDB-dashboard/blob/main/README.md
  4. Add the extract found at https://raw.githubusercontent.com/RamSailopal/YottaDB-dashboard/main/prometheus.yml to the prometheus.yml config file in /etc/prometheus. Change the IP address and port to the address/port of the server where the “exporter” is running accordingly.
  5. Start Prometheus.
  6. Start Grafana and log into the web interface (default port is 3000, default username and password is admin/admin).
  7. Click on the cog icon on the left hand side representing configuration, then Data Sources and Add Data Source. Select the Prometheus data source. Name it and then enter the URL. Click Save and Test to ensure it is working.
  8. Click on the four square boxes icon on the left hand side representing Dashboards. Click Manage and then Import. Paste the JSON found here – https://raw.githubusercontent.com/RamSailopal/yotta-dashboard/main/examples/Yottadb-grafana.json to the json panel. Click Load.
  9. A dashboard will then be created showing Orders, Sets and Gets metrics. To add further metrics, click on the add panel icon, then click the Metrics “drop down” option in the Query section, ensuring that Prometheus is selected as the data source. All the metrics exported from YottaDB should now appear for selection to chart in a panel.

The Grafana/Prometheus technology stack, be it on physical servers or containers, can further be harnessed to view memory/CPU consumption along-side software metrics as discussed earlier. Additionally, it can be used to monitor database size. System metrics relevant to YottaDB are available as YottaDB-dashboard metrics. The technology stack can also be further added to, by incorporating Grafana Promtail and Loki components to view and interrogate YottaDB system logs. Anybody looking for further information can visit the YottaDB-dashboard Wiki.

Conclusion

YottaDB has open source at its core. This embracing of open source, allows it to be integrated with other open source tools to further extend its capabilities and benefit a wider range of users operating in different areas of IT operations.

The YottaDB-dashboard also has tools and information to help you provision a virtual machine with the YottaDB + Prometheus + Grafana stack using Ansible and Vagrant.

About Raman Sailopal

I am a professional with over 20 years of experience working in the IT industry. I reside in Birmingham, UK (hence the Birmingham themed pictures). My very first corporate interaction with IT involved using M technology within the UK medical sector and although my journey with IT has seen me follow paths involving Linux, Cloud architecture and open source, I have always held a fond regard for M based technologies and their potential with critical IT systems. Working with YottaDB allows me to combine these passions and contribute to the growth of both M and open source.

 

 

 

Images Used:
[1] Young man taking the Peaky Blinders soundtrack album out of his vinyl collection. Photo by Jonas Leupe.
[2] The Canal House, Birmingham. Photo by Danilo D’Agostino.
Screenshots courtesy Raman Sailopal.

Building Rich Web Applications with YottaDB

We thank Rob Tweed for the first guest post on the YottaDB blog and hope there are many more to follow. If you would like to post on the YottaDB blog, please contact us at info@yottadb.com.

Background

YottaDB is the perfect database to support Web Applications:

  • It is hierarchical key-value, “global variable”, storage model is incredibly flexible, allowing any database architecture to be modeled on top, including:
    • relational (with SQL support);
    • JSON/document;
    • lists, stacks and/or queues;
    • graph;
    • tables and/or columnar stores; and
    • Document Object Model (e.g., XML with XPath support).
  • It is Free / Open Source database technology.
  • It is incredibly fast, with key/value storage operating at or near in-memory speeds.
  • It is highly scalable and robust.
  • It is tried and tested, with a long and respected pedigree spanning decades, supporting, amongst other things, the highly demanding needs of large-scale, commercial, real-time banking systems.

For YottaDB to be a Web Application database, it must be integrated with a Web Server. The Web Server provides the outward-facing HTTP/HTTPS interface by which web browsers and REST clients communicate with the web application server.

Over the years, there have been numerous solutions for web-enablement of YottaDB, including:

  • CGI or FastCGI solutions;
  • the MGWSI interface from M/Gateway Development Ltd; and
  • a web server written in the M language.

M/Gateway Developments Ltd have recently introduced new solutions:

  • the low-level mg_web integration interface;
  • mgweb-server, a high-level platform, built on top of mg_web that provides a pre-packaged solution for quickly and easily creating a YottaDB-based API Server.

This post looks in more detail at mg_web and mgweb-server to examine their benefits over the existing solutions for YottaDB.

Why mg_web?

Since YottaDB can already be configured as an effective and potent web application platform using existing solutions, why consider the new M/Gateway solutions? Chris Munt, mg_web’s developer, had a number of design goals:

  • Leveraging industry-standard Web Servers: Web Servers are no longer the simple HTTP-enabling technology they used to be. They are now very complex pieces of technology. The overwhelming majority of web traffic on both the Internet and within organizations is handled by just three Web Servers:

All three are well understood by their respective users, and each is more than capable of meeting the needs of modern web applications and API servers where performance, reliability, scalability and security are key. Chris believes that it makes sense to use these industrial-strength web servers for your industrial-strength, mission-critical applications. mg_web works with all three web servers.

  • Exploiting the latest Web Server architectures: Most of the techniques used for web-enablement of YottaDB were first developed when the web was in its infancy. Since then, all three major Web Servers have moved to ultra-high performance, event-driven, asynchronous architectures which the existing solutions for YottaDB fail to exploit. By contrast, Chris took the opportunity to design mg_web from the ground up to work with and exploit these latest design features of Web Servers.
  • Meeting the emerging needs of the modern web: Two features of Web Servers are growing in importance:
    • Security: HTTPS over TLS (*aka* SSL) is likely to become mandatory soon, and Web Servers need to be able to support a growing number of security-related features and specialised HTTP request and response headers. By working with all three major Web Servers and allowing them to handle the complexities of these ever-changing and evolving security needs, Chris designed mg_web to future-proof your web-enabled YottaDB service.
    • Performance: The simple text-based protocol used for REST and web traffic is increasingly being replaced by the newer HTTP/2 protocol. HTTP/2 brings some key benefits, the main one being the improvement of page loading times by:
      • The use of a binary transfer format instead of the previous text format
      • Multiplexing of requests for resources, meaning, for example, that a browser can fetch multiple resources (i.e. HTML, CSS, JavaScript files) simultaneously via a single TCP connection rather than, as previously, having to open multiple TCP connections
      • Compression of header frames to reduce the size of transmitted payloads

Implementing the much more complex and demanding HTTP/2 protocol is something best left to the major Web Server suppliers, for whom it’s core business. Chris ensured that mg_web fully supports HTTP/2, meaning your YottaDB web applications and API Services can take advantage of the benefits of HTTP/2 immediately.

  • Supporting the move from server-side to client side applications: Early web-enablement of YottaDB applications focused around server-side logic that dynamically generated web pages served up to users’ browsers. Since then, the industry has almost completely evolved to client-side applications where JavaScript-based logic runs in the browser and the role of the back-end is now primarily to serve up data in the form of JSON-formatted responses: the so-called REST API Server role. The existing YottaDB solutions don’t actually tend to be optimised for such a role. Chris designed mg_web to work optimally in this new, modern API server role, while ensuring that it could also support any older-style server-side applications. This provides the best of both worlds, allowing you to evolve and migrate your legacy applications to a modern design at your own pace.
  • Achieving new levels of performance with YottaDB: Existing solutions for YottaDB when using the major Web Servers have made use of networked connections between the Web Server and YottaDB. Whilst mg_web still allows such a style of connection, Chris also allows your application to use YottaDB’s ultra-high-performance in-process C API.

Public versus In-house Web Applications

Most people probably associate web applications with the public Internet where they have become the de-facto standard for application design and delivery. However, from the earliest days of the web, many realised that the very same architecture and technologies could be used to create a highly effective and cost-effective way for designing and delivering private, in-house applications, made available across an organisation’s internal network.

Whilst leveraging the same web technologies, a key difference between a publicly-available web application and a private, locally-networked one is security. A publicly-accessibly web application must be carefully designed to prevent its architectural components from being exploited as a means of gaining unauthorised access to the underlying database: it’s probably the single most important aspect of its design. The scalability of a publicly-accessible web application is also of key importance: you may need multiple, parallel database instances to spread the load in times of heavy demand.

In contrast, a private web application, limited to access across an internal network, is less prone to unauthorised access from outside the organisation (assuming, of course, adequate controls are in place), and a key focus is performance. In this scenario you will have more control over the peak levels of demand for the web service as its user-base will be limited to authorised users within the organisation.

mg_web has been designed with these differences in mind, to allow the optimal use of YottaDB in either, or both, scenarios. As outlined in the previous section above, you can configure mg_web to use:

  • Networked access to one or more YottaDB servers, allowing the most secure and scalable configuration. The public-facing part of web application is the web server, and mg_web in networked mode isolates the web server from the database.
  • API access YottaDB servers, allows the highest levels of performance, with a closer and more intimate connection between the web server and YottaDB, more suited to an in-house, private application.

mg_web even allows you to mix and match these: it makes sure that you are in control, and you can decide the best and most appropriate mix of performance and security for your particular requirements.

Legacy Web Applications versus Modern REST-based API Servers

There are many legacy, older, server-side applications web applications. mg_web is the ideal solution to allow you to quickly and easily migrate those legacy applications to a more modern technology that takes advantage of its many benefits, without having to modify much, if any, of core application logic: mg_web runs the M code of those legacy applications!

Applications you build in the future will likely be designed as client-based web applications that make REST requests over HTTPS to get and set YottaDB data. I have created a higher-level platform called mgweb-server that manages setting up and configuring an mg_web-based REST API server. For example, mgweb-server:

  • Includes all the basic configuration settings for an Apache-based mg_web system, and all the configuration files for integrating with YottaDB over either a network or API connection.
  • Provides a straightforward pattern for defining your REST API “routes” – i.e., defining the REST request patterns that you want to accept, and mapping them to your corresponding handler functions.
  • Implements JSON data exchange between client and server. It includes functions that convert bi-directionally between JSON and YottaDB local variables (process-private hierarchical key-value data). Your application logic can simply manipulate local variable (sub)trees, and mgweb-server will convert your local variable to and from JSON.
  • Includes APIs for securely storing and authenticating user passwords.
  • Includes APIs for creating, authenticating, decoding and modifying JSON Web Tokens (JWTs). Modern REST-based API services increasingly use JWTs as their preferred means of security, user authentication and state management.

mgweb-server comes in two flavours:

  • A pre-built Dockerised version that includes pre-installed and configured instances of the Apache Web Server and YottaDB. This version is not just the quickest and easiest way to try out *mgweb-server* and mg_web, but is also a production-ready platform, saving you time and effort. This Dockerised can also be easily re-configured to integrate with any of your existing YottaDB server(s) instead of (or as well as) using the pre-installed version of YottaDB within the Container.
  • A “build it yourself” kit that contains all the configuration files and API routine files to customize your own mgweb-server REST API Server platform that uses your own Apache Web Server(s) and your own YottaDB database(s). Before using this flavour, you should first familiarise yourself with mgweb-server using the Dockerised version.

Conclusion

Between mg_web and mgweb-server, M/Gateway Developments ensure you have all options catered for. You can retain and future-proof your existing legacy of web applications, whilst also being able to use the same underlying mg_web platform with mgweb-server as a polished and easy solution for building your new REST API server-based applications. You can configure your Web Server to work with any number of YottaDB databases, and you can choose between the more flexible networked connections or the high-speed in-process connection between the web server and YottaDB.

Both mg_web and mgweb-server are Apache 2.0 licensed open source software, so there really is no reason not to adopt them for your YottaDB web application development.

About Rob Tweed

Rob Tweed is a software developer, consultant, and teacher who has been active in the M community from long before the days when the M database was unbundled from the M language. Rob and his business partner Chris Munt were pioneers in exploiting emerging web technologies to create rich user interfaces for server-based applications. While software is his vocation, photography is Rob’s avocation, and he is equally passionate about both. The photographs accompanying this post and on the blog page linking to this blog post appear courtesy Rob Tweed. You will find more of his pictures at https://instagram.com/robtweed1955. Thanks, Rob!