Blog
Sorry, your browser does not support inline SVG.

Building Rich Web Applications with YottaDB

Rob Tweed

We thank Rob Tweed for the first guest post on the YottaDB blog and hope there are many more to follow. If you would like to post on the YottaDB blog, please contact us at info@yottadb.com.

Background

YottaDB is the perfect database to support Web Applications:

  • It is hierarchical key-value, “global variable”, storage model is incredibly flexible, allowing any database architecture to be modeled on top, including:
    • relational (with SQL support);
    • JSON/document;
    • lists, stacks and/or queues;
    • graph;
    • tables and/or columnar stores; and
    • Document Object Model (e.g., XML with XPath support).
  • It is Free / Open Source database technology.
  • It is incredibly fast, with key/value storage operating at or near in-memory speeds.
  • It is highly scalable and robust.
  • It is tried and tested, with a long and respected pedigree spanning decades, supporting, amongst other things, the highly demanding needs of large-scale, commercial, real-time banking systems.

For YottaDB to be a Web Application database, it must be integrated with a Web Server. The Web Server provides the outward-facing HTTP/HTTPS interface by which web browsers and REST clients communicate with the web application server.

Over the years, there have been numerous solutions for web-enablement of YottaDB, including:

  • CGI or FastCGI solutions;
  • the MGWSI interface from M/Gateway Development Ltd; and
  • a web server written in the M language.

M/Gateway Developments Ltd have recently introduced new solutions:

  • the low-level mg_web integration interface;
  • mgweb-server, a high-level platform, built on top of mg_web that provides a pre-packaged solution for quickly and easily creating a YottaDB-based API Server.

This post looks in more detail at mg_web and mgweb-server to examine their benefits over the existing solutions for YottaDB.

Why mg_web?

Since YottaDB can already be configured as an effective and potent web application platform using existing solutions, why consider the new M/Gateway solutions? Chris Munt, mg_web’s developer, had a number of design goals:

  • Leveraging industry-standard Web Servers: Web Servers are no longer the simple HTTP-enabling technology they used to be. They are now very complex pieces of technology. The overwhelming majority of web traffic on both the Internet and within organizations is handled by just three Web Servers:

All three are well understood by their respective users, and each is more than capable of meeting the needs of modern web applications and API servers where performance, reliability, scalability and security are key. Chris believes that it makes sense to use these industrial-strength web servers for your industrial-strength, mission-critical applications. mg_web works with all three web servers.

  • Exploiting the latest Web Server architectures: Most of the techniques used for web-enablement of YottaDB were first developed when the web was in its infancy. Since then, all three major Web Servers have moved to ultra-high performance, event-driven, asynchronous architectures which the existing solutions for YottaDB fail to exploit. By contrast, Chris took the opportunity to design mg_web from the ground up to work with and exploit these latest design features of Web Servers.
  • Meeting the emerging needs of the modern web: Two features of Web Servers are growing in importance:
    • Security: HTTPS over TLS (*aka* SSL) is likely to become mandatory soon, and Web Servers need to be able to support a growing number of security-related features and specialised HTTP request and response headers. By working with all three major Web Servers and allowing them to handle the complexities of these ever-changing and evolving security needs, Chris designed mg_web to future-proof your web-enabled YottaDB service.
    • Performance: The simple text-based protocol used for REST and web traffic is increasingly being replaced by the newer HTTP/2 protocol. HTTP/2 brings some key benefits, the main one being the improvement of page loading times by:
      • The use of a binary transfer format instead of the previous text format
      • Multiplexing of requests for resources, meaning, for example, that a browser can fetch multiple resources (i.e. HTML, CSS, JavaScript files) simultaneously via a single TCP connection rather than, as previously, having to open multiple TCP connections
      • Compression of header frames to reduce the size of transmitted payloads

Implementing the much more complex and demanding HTTP/2 protocol is something best left to the major Web Server suppliers, for whom it’s core business. Chris ensured that mg_web fully supports HTTP/2, meaning your YottaDB web applications and API Services can take advantage of the benefits of HTTP/2 immediately.

  • Supporting the move from server-side to client side applications: Early web-enablement of YottaDB applications focused around server-side logic that dynamically generated web pages served up to users’ browsers. Since then, the industry has almost completely evolved to client-side applications where JavaScript-based logic runs in the browser and the role of the back-end is now primarily to serve up data in the form of JSON-formatted responses: the so-called REST API Server role. The existing YottaDB solutions don’t actually tend to be optimised for such a role. Chris designed mg_web to work optimally in this new, modern API server role, while ensuring that it could also support any older-style server-side applications. This provides the best of both worlds, allowing you to evolve and migrate your legacy applications to a modern design at your own pace.
  • Achieving new levels of performance with YottaDB: Existing solutions for YottaDB when using the major Web Servers have made use of networked connections between the Web Server and YottaDB. Whilst mg_web still allows such a style of connection, Chris also allows your application to use YottaDB’s ultra-high-performance in-process C API.

Public versus In-house Web Applications

Most people probably associate web applications with the public Internet where they have become the de-facto standard for application design and delivery. However, from the earliest days of the web, many realised that the very same architecture and technologies could be used to create a highly effective and cost-effective way for designing and delivering private, in-house applications, made available across an organisation’s internal network.

Whilst leveraging the same web technologies, a key difference between a publicly-available web application and a private, locally-networked one is security. A publicly-accessibly web application must be carefully designed to prevent its architectural components from being exploited as a means of gaining unauthorised access to the underlying database: it’s probably the single most important aspect of its design. The scalability of a publicly-accessible web application is also of key importance: you may need multiple, parallel database instances to spread the load in times of heavy demand.

In contrast, a private web application, limited to access across an internal network, is less prone to unauthorised access from outside the organisation (assuming, of course, adequate controls are in place), and a key focus is performance. In this scenario you will have more control over the peak levels of demand for the web service as its user-base will be limited to authorised users within the organisation.

mg_web has been designed with these differences in mind, to allow the optimal use of YottaDB in either, or both, scenarios. As outlined in the previous section above, you can configure mg_web to use:

  • Networked access to one or more YottaDB servers, allowing the most secure and scalable configuration. The public-facing part of web application is the web server, and mg_web in networked mode isolates the web server from the database.
  • API access YottaDB servers, allows the highest levels of performance, with a closer and more intimate connection between the web server and YottaDB, more suited to an in-house, private application.

mg_web even allows you to mix and match these: it makes sure that you are in control, and you can decide the best and most appropriate mix of performance and security for your particular requirements.

Legacy Web Applications versus Modern REST-based API Servers

There are many legacy, older, server-side applications web applications. mg_web is the ideal solution to allow you to quickly and easily migrate those legacy applications to a more modern technology that takes advantage of its many benefits, without having to modify much, if any, of core application logic: mg_web runs the M code of those legacy applications!

Applications you build in the future will likely be designed as client-based web applications that make REST requests over HTTPS to get and set YottaDB data. I have created a higher-level platform called mgweb-server that manages setting up and configuring an mg_web-based REST API server. For example, mgweb-server:

  • Includes all the basic configuration settings for an Apache-based mg_web system, and all the configuration files for integrating with YottaDB over either a network or API connection.
  • Provides a straightforward pattern for defining your REST API “routes” – i.e., defining the REST request patterns that you want to accept, and mapping them to your corresponding handler functions.
  • Implements JSON data exchange between client and server. It includes functions that convert bi-directionally between JSON and YottaDB local variables (process-private hierarchical key-value data). Your application logic can simply manipulate local variable (sub)trees, and mgweb-server will convert your local variable to and from JSON.
  • Includes APIs for securely storing and authenticating user passwords.
  • Includes APIs for creating, authenticating, decoding and modifying JSON Web Tokens (JWTs). Modern REST-based API services increasingly use JWTs as their preferred means of security, user authentication and state management.

mgweb-server comes in two flavours:

  • A pre-built Dockerised version that includes pre-installed and configured instances of the Apache Web Server and YottaDB. This version is not just the quickest and easiest way to try out *mgweb-server* and mg_web, but is also a production-ready platform, saving you time and effort. This Dockerised can also be easily re-configured to integrate with any of your existing YottaDB server(s) instead of (or as well as) using the pre-installed version of YottaDB within the Container.
  • A “build it yourself” kit that contains all the configuration files and API routine files to customize your own mgweb-server REST API Server platform that uses your own Apache Web Server(s) and your own YottaDB database(s). Before using this flavour, you should first familiarise yourself with mgweb-server using the Dockerised version.

Conclusion

Between mg_web and mgweb-server, M/Gateway Developments ensure you have all options catered for. You can retain and future-proof your existing legacy of web applications, whilst also being able to use the same underlying mg_web platform with mgweb-server as a polished and easy solution for building your new REST API server-based applications. You can configure your Web Server to work with any number of YottaDB databases, and you can choose between the more flexible networked connections or the high-speed in-process connection between the web server and YottaDB.

Both mg_web and mgweb-server are Apache 2.0 licensed open source software, so there really is no reason not to adopt them for your YottaDB web application development.

About Rob Tweed

Rob Tweed is a software developer, consultant, and teacher who has been active in the M community from long before the days when the M database was unbundled from the M language. Rob and his business partner Chris Munt were pioneers in exploiting emerging web technologies to create rich user interfaces for server-based applications. While software is his vocation, photography is Rob’s avocation, and he is equally passionate about both. The photographs accompanying this post and on the blog page linking to this blog post appear courtesy Rob Tweed. You will find more of his pictures at https://instagram.com/robtweed1955. Thanks, Rob!

 

Published on March 05, 2021

See All General

Back to Blog