One of the strengths of SciDB over other database management systems is its extensibility.1 SciDB allows the user to add new data types, functions, and operators. In this multi-part post, we discuss various aspects of extending SciDB. In this post we look at the available documentation and how to setup the development tools.

Documentation on Extending SciDB

The official documentation on extending SciDB is at best sparse. There is some discussion in this SciDB tutorial from 2013. (We talk more about tutorials in SciDB in this post.) Older SciDB versions include a special chapter in the documentation on User-Defined Types and Functions. The most recent such chapter is for SciDB version 14.8 and is available here. More recent SciDB versions do not contain such a chapter in the documentation. This documentation chapter can provide the reader with an overview of what it takes to create user defined type and functions.

Another useful piece of documentation is the documentation embedded in the SciDB code. This documentation is not officially supported by Paradigm4, but can be generated from the source code using the Doxygen documentation generator. This can be done right after completing the Building SciDB CE step of the official SciDB Community Edition Installation Guide. The following sequence of commands can be used:

> cd <dev_dir>/scidbtrunk
> echo 'add_subdirectory("doc")' >> CMakeLists.txt
> ./run.py setup
> cd stage/build
> make doc
> ls doc/api/html/index.html

Note that the Doxygen package needs to be installed first. The SciDB code documentation can now be browsed starting for the index.html file listed above.

Finally, the best available “documentation” on extending SciDB are the existing plugins provided by Paradigm4 on their GitHub page or third-party developers. One special such plugin is the dev_tools plugin (see GitHub). We discuss this plugin next.

Installing the Development Tools

One of the SciDB plugins provided by Paradigm4 on their GitHub page is the dev_tools plugin (see GitHub). Once installed, this plugin allows the user to install other SciDB plugins directly from GitHub. The plugin comes with a detailed README file on how to install this plugin for different SciDB versions and Linux distributions. We highly recommend installing this plugin first. Once installed, we can load it and verify the available operators using:

# iquery --afl
AFL% load_library('dev_tools');
Query was executed successfully
AFL% filter(list('operators'), library = 'dev_tools');
{No} name,library
{21} 'install_github','dev_tools'

Notice the newly available operator install_github. See the Synopsis and the Example provided in the plugin README file for more information on how to use this operator. Now, we can use this operator to install other SciDB plugins from GitHub. A good plugin to start with is the limit plugin (see GitHub) also provided by Paradigm4. This is a good minimal plugin which can be used as a skeleton for new plugins and guidance on how to organize the code. We can install, load, and verify the available operators using:

# iquery --afl
AFL% install_github('paradigm4/limit');
{i} success
{0} true
AFL% load_library('limit');
Query was executed successfully
AFL% filter(list('operators'), library = 'limit');
{No} name,library
{24} 'limit','limit'

The steps and queries used in this post are available here, as follows:

1 One notable exception is PostgreSQL which was also lead by Michael Stonebraker. ↩