RocksDB
{{Short description|Embedded key-value database}}
{{Infobox software
| name = RocksDB
| logo = Rocksdb-icon.svg
| screenshot =
| caption = Embedded persistent key/value store.
| collapsible =
| author = Dhruba Borthakur
| developer = Meta Platforms (was Facebook, Inc.)
| released = {{Start date and age|2012|05}}
| latest release version = {{wikidata|property|edit|reference|P548=Q2804309|P348}}
| latest release date = {{wikidata|qualifier|P548=Q2804309|P348|P577}}
| programming language = C++
| operating system = Windows, macOS, Linux, FreeBSD, OpenBSD, Solaris, AIX
| platform = Cross-platform
| genre = Embedded database
| license = Apache 2.0 or GPL 2
| website = {{URL|rocksdb.org}}
}}
RocksDB is a high performance{{cite web |url=https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks |title=Performance Benchmarks |website=GitHub |access-date=November 29, 2015}}{{cite web |url=http://smalldatum.blogspot.com/2014/07/benchmarking-leveldb-family.html |title=Benchmarking the leveldb family |date=7 July 2014 |access-date=March 10, 2016}}{{cite web |url=http://smalldatum.blogspot.com/2015/04/comparing-leveldb-and-rocksdb-take-2.html |title=Comparing LevelDB and RocksDB, take 2 |date=27 April 2015 |access-date=March 10, 2016}}{{cite web |url=https://influxdata.com/blog/benchmarking-leveldb-vs-rocksdb-vs-hyperleveldb-vs-lmdb-performance-for-influxdb/ |title=Benchmarking LevelDB vs. RocksDB vs. HyperLevelDB vs. LMDB Performance for InfluxDB |date=20 June 2014 |access-date=March 10, 2016}}{{cite book |last1=Golan-Gueta |first1=Guy |last2=Bortnikov |first2=Edward |last3=Hillel |first3=Eschar |last4=Keidar |first4=Idit |author4-link=Idit Keidar|title=Proceedings of the Tenth European Conference on Computer Systems |chapter=Scaling concurrent log-structured data stores |date=April 21, 2015 |doi=10.1145/2741948.2741973 |pages=1–14 |isbn=9781450332385 |s2cid=5849146}} embedded database for key-value data. It is a fork of Google's LevelDB optimized to exploit multi-core processors (CPUs), and make efficient use of fast storage, such as solid-state drives (SSD), for input/output (I/O) bound workloads. It is based on a log-structured merge-tree (LSM tree) data structure. It is written in C++ and provides official language bindings for C++, C, and Java. Many third-party language bindings exist. RocksDB is free and open-source software, released originally under a BSD 3-clause license.{{cite web |url=https://gigaom.com/2013/11/21/facebooks-latest-open-source-effort-a-flash-powered-database-called-rocksdb/ |title=Facebook's latest open source effort: a flash-powered database called RocksDB |date=21 November 2013 |access-date=March 10, 2016 |archive-date=24 February 2020 |archive-url=https://web.archive.org/web/20200224065917/https://gigaom.com/2013/11/21/facebooks-latest-open-source-effort-a-flash-powered-database-called-rocksdb/ |url-status=dead }}{{cite web |url=https://www.facebook.com/notes/facebook-engineering/under-the-hood-building-and-open-sourcing-rocksdb/10151822347683920/ |title=Under the Hood: Building and open-sourcing RocksDB |website=Facebook |access-date=March 10, 2016}}{{cite web |url=http://www.i-programmer.info/news/84-database/6624-rocksdb-facebooks-database-now-open-source.html |title=RocksDB - Facebook's Database Now Open Source |access-date=March 10, 2016}} However, in July 2017 the project was migrated to a dual license of both Apache 2.0 and GPLv2 license.{{cite web |url=https://github.com/facebook/rocksdb/tree/3c327ac2d0fd50bbd82fe1f1af5de909dad769e6/env |title=GitHub pull request |website=GitHub |access-date=July 20, 2017}} This change helped its adoption in Apache Software Foundation's projects after blacklist of the previous BSD+Patents license clause.{{cite web |url=https://www.theregister.co.uk/2017/07/17/apache_says_no_to_facebook_code_libraries/ |title=Apache says 'no' to Facebook code libraries
|website=The Register
|access-date=July 20, 2017}}{{cite web |url=https://github.com/facebook/rocksdb/issues/2605 |title=GitHub issue |website=GitHub |access-date=July 20, 2017}}
RocksDB is used in production systems at various web-scale enterprises{{cite web |url=https://github.com/facebook/rocksdb/blob/master/USERS.md |title=Users.md |website=GitHub |access-date=December 1, 2015}} including Facebook, Yahoo!,{{cite web |url=http://www.i-programmer.info/news/84-database/8542-rocksdb-on-steroids.html |title=RocksDB on Steroids |access-date=March 10, 2016}} and LinkedIn.{{cite web |url=https://engineering.linkedin.com/performance/benchmarking-apache-samza-12-million-messages-second-single-node |title=Benchmarking Apache Samza: 1.2 million messages per second on a single node |access-date=March 10, 2016}}
Features
RocksDB, like LevelDB, stores keys and values in arbitrary byte arrays, and data is sorted byte-wise by key or by providing a custom comparator.
RocksDB provides all of the features of LevelDB, plus:
- Transactions{{Cite web |url=https://github.com/facebook/rocksdb/wiki/Transactions |title=RocksDB transactions |website=GitHub|access-date=2016-04-04}}
- Backups{{Cite web |url=https://github.com/facebook/rocksdb/wiki/How-to-backup-RocksDB |title=How to backup RocksDB? |website=GitHub|access-date=2017-07-19}} and snapshots{{Cite web |url=https://github.com/facebook/rocksdb/wiki/Checkpoints |title=Checkpoints |website=GitHub|access-date=2017-07-19}}
- Column families{{Cite web |url=https://github.com/facebook/rocksdb/wiki/Column-Families |title=Column families in RocksDB |website=GitHub|access-date=2016-04-04}}
- Bloom filters{{Cite web |url=https://github.com/facebook/rocksdb/wiki/RocksDB-Bloom-Filter |title=RocksDB bloom filters |website=GitHub|access-date=2016-04-04}}
- Time to live (TTL) support{{Cite web |url=https://github.com/facebook/rocksdb/wiki/Time-to-Live |title=RocksDB TTL support |website=GitHub|access-date=2016-04-04}}
- Universal compaction{{Cite web |url=https://github.com/facebook/rocksdb/wiki/Universal-Compaction |title=Universal compaction |website=GitHub|access-date=2016-04-04}}
- Merge operators{{Cite web |url=https://github.com/facebook/rocksdb/wiki/Merge-Operator-Implementation |title=RocksDB merge operator |website=GitHub|access-date=2016-04-04}}
- Statistics collection{{Cite web |url=https://github.com/facebook/rocksdb/wiki/Perf-Context-and-IO-Stats-Context |title=RocksDB perf context and IO stats context |website=GitHub|access-date=2016-04-04}}
- Geospatial indexing{{Cite web |url=https://rocksdb.org/blog/2015/07/17/spatial-indexing-in-rocksdb.html |title=Spatial indexing in RocksDB |website=rocksdb.org|access-date=2018-07-19}}
RocksDB is not an SQL database (although MyRocks combines RocksDB with MySQL). Like other NoSQL and dbm stores, it has no relational data model, and it does not support SQL queries. Also, it has no direct support for secondary indexes, however a user may build their own internally using Column Families or externally. Applications use RocksDB as a library, as it provides no server or command-line interface.
History
RocksDB was created at Facebook by Dhruba Borthakur{{cite web |url= https://github.com/facebook/rocksdb/commit/cc6c32535aec596036c56fe742a39182539f76fa |title=First commit where RocksDB diverges from LevelDB |website=GitHub |date=May 10, 2012 |access-date=March 15, 2016}}{{cite web |url=https://github.com/facebook/rocksdb/commit/6eb5ed9a4922ec2d121768adfa3ab3b6ead5d627 |title=Rocksdb readme file |website=GitHub |date=Nov 30, 2012 |access-date=March 15, 2016}} in April 2012, as a fork of LevelDB with the initial stated goal of improving performance for server workloads.{{cite web |url=http://rocksdb.blogspot.com/2013/11/the-history-of-rocksdb.html |title=The History of RocksDB |date= November 24, 2013 |access-date=March 10, 2016}}{{cite web |url=https://www.youtube.com/watch?v=V_C-T5S-w8g |title=RocksDB: A High Performance Embedded Key-Value Store for Flash Storage - Data@Scale |date= November 22, 2013 |access-date=March 10, 2016 |quote=... The story of why we decided to do RocksDB ... |first1=Dhruba |last1=Borthakur |website=YouTube}}
Integration
As an embeddable database, RocksDB can be used as a storage engine within a larger database management system (DBMS). For example, Rockset uses RocksDB{{cite web |url=https://rockset.com/blog/how-we-use-rocksdb-at-rockset/ |first1=Sandeep |last1=Dhoot |date=2019-06-27 |accessdate=2023-03-01 |title=How We Use RocksDB at Rockset |website=rockset.com}} mostly for analytical data processing.
= Alternative backend =
The following projects have been started to replace or offer alternative storage engines for already-established database systems with RocksDB:
== ArangoDB ==
ArangoDB has added RocksDB to its previous storage engine ("mmfiles").{{Cite web|url=https://arangodb.com/community-server/rocksdb-storage-engine/|title=Comparing new RocksDB and MMFiles storage engines|website=ArangoDB}} RocksDB is be the default storage engine since ArangoDB 3.4.{{cite web |url=https://www.arangodb.com/2018/09/rc1-arangodb-3-4-whats-new/ |title=RC1 ArangoDB 3.4 - Whats new? |date=6 September 2018}}
== Cassandra ==
Cassandra on RocksDB can improve the performance of Apache Cassandra significantly (3–4 times faster in general, 100 times faster in some use-cases).{{citation needed |date=May 2020}} The Instagram team at Facebook developed and open-sourced their code, along with benchmarks of their performance results.{{cite web |url=https://instagram-engineering.com/open-sourcing-a-10x-reduction-in-apache-cassandra-tail-latency-d64f86b43589 |title=Open-sourcing a 10x reduction in Apache Cassandra tail latency |date=5 March 2018}}
== MariaDB ==
MariaDB can use the MyRocks storage engine (which is forked from RocksDB) since MariaDB 10.2.5 (Alpha status) {{Cite web |url=https://mariadb.com/kb/en/library/myrocks/ |title=MyRocks |website=MariaDB KnowledgeBase|access-date=2019-04-28}} and stable since MariaDB 10.2.16 in 2018.{{Cite web|url=https://mariadb.com/kb/en/mariadb-10216-release-notes/|title=MariaDB 10.2.16 Release Notes|website=MariaDB KnowledgeBase}}
== MongoDB ==
The MongoRocks project provides a storage module for MongoDB where the storage engine is RocksDB.{{cite web |url=https://github.com/mongodb-partners/mongo-rocks |title=mongodb-partners/mongo-rocks |website=GitHub |date=29 October 2021}}{{cite web |url=https://rocksdb.org/blog/2015/04/22/integrating-rocksdb-with-mongodb-2.html |title=Integrating RocksDB with MongoDB |access-date=July 19, 2018}}{{cite web |url=http://blog.parse.com/announcements/mongodb-rocksdb-parse/ |title=MongoDB + RocksDB at Parse |access-date=December 1, 2015}}
A related program is Rocks Strata, a tool written in Go, which allows managing incremental backups of MongoDB when RocksDB is used as the storage engine.{{cite web |url=https://github.com/facebookgo/rocks-strata |title=facebookgo/rocks-strata |website=GitHub |date=31 October 2021}}
== MySQL ==
The MyRocks project created a new RocksDB-based storage engine for MySQL.{{cite web |url=https://github.com/facebook/mysql-5.6 |title=facebook/mysql-5.6 |website=GitHub |date=2 November 2021}}{{cite web |url=https://www.percona.com/live/europe-amsterdam-2015/sites/default/files/slides/MyRocksAtFacebook_201509.pdf |title=MyRocks: MySQL on RocksDB |access-date=November 29, 2015}} In-depth details about MyRocks were presented at Percona Live 2016.{{cite web |url=http://www.slideshare.net/matsunobu/myrocks-deep-dive/ |title=MyRocks Deep Dive |date=19 April 2016 |access-date=May 9, 2016}}
== Oxigraph ==
Oxigraph{{Cite web|url=https://github.com/oxigraph/oxigraph|title=Oxigraph|first=Thomas|last=Pellissier Tanon|date=July 12, 2024|via=GitHub}} is a graph database implementing the SPARQL standard, based on RocksDB
== UKV ==
The UKV{{cite web |url=https://github.com/unum-cloud/ukv |title=unum-cloud/ukv |website=GitHub |date=28 December 2022}} project allows users to use RocksDB on par with LevelDB as the underlying key-value store. It represents a shared abstraction for create, read, update and delete (CRUD) operations common to every storage engine. It augments it with structured bindings for several high-level languages, including Python, Java, and Go.
= Embedded =
The following database systems and applications have chosen to use RocksDB as their embedded storage engine:
== Ceph's BlueStore ==
The Ceph's BlueStore storage layer uses RocksDB for metadata management in OSD devices.{{cite web |url=http://docs.ceph.com/docs/luminous/rados/configuration/storage-devices/ |title=Storage Devices -- Ceph Documentation |access-date=2017-11-08 |archive-date=2020-02-24 |archive-url=https://web.archive.org/web/20200224065932/https://docs.ceph.com/docs/luminous/rados/configuration/storage-devices/ |url-status=dead }}
== Apache Flink ==
== FusionDB ==
FusionDB{{cite web |url=https://fusiondb.com |title=FusionDB|publisher=Evolved Binary}} uses RocksDB as its storage engine for XML, Key/Value, and JSON.{{cite web |url=https://archive.xmlprague.cz/2019/files/xmlprague-2019-proceedings.pdf#page179 |title=The Design and Implementation of FusionDB|publisher=XML Prague}}
== LogDevice LogsDB ==
== Kafka Streams ==
== Manhattan ==
The Manhattan Distributed Key-Value Store has used RocksDB as its primary engine to store Twitter data since 2018.{{cite web |url=https://blog.twitter.com/engineering/en_us/topics/infrastructure/2021/adopting-rocksdb-within-manhattan |title=Adopting RocksDB within Manhattan |website=Twitter |date=28 December 2022}}
== Rockset ==
The Rockset{{Cite web|url=https://rockset.com/|title=Rockset: Search and analytics database|website=rockset.com}} service that is used for operational data analytics uses RocksDB as its storage engine.{{Cite web |url=https://rockset.com/blog/how-we-use-rocksdb-at-rockset/ |title=How we use RocksDB at Rockset |website=rockset.com|language=en|access-date=2019-07-10}}
== SSDB ==
The ssdb-rocks{{cite web |url=https://github.com/ideawu/ssdb-rocks |title=ideawu/ssdb-rocks |website=GitHub |date=21 August 2021}} project uses RocksDB as the storage engine for the SSDB{{Cite web|url=https://ssdb.io/|title=Home|website=ID-2Sbo}} NoSQL Database.
== TiDB ==
The TiDB{{cite web |url=https://github.com/pingcap/tidb |title=pingcap/tidb |website=GitHub |date=4 November 2021}} project uses RocksDB as its storage engine.{{cite web |url=https://pingcap.com/blog/2017-07-11-tidbinternal1 |title=TiDB Internal (I) - Data Storage |date=11 July 2017 |publisher=Shen Li}}
== YugabyteDB ==
The YugabyteDB database uses a modified version of RocksDB as part of its DocDB storage engine.{{cite web |date=20 February 2019 |title=How We Built a High Performance Document Store on RocksDB? |url=https://blog.yugabyte.com/how-we-built-a-high-performance-document-store-on-rocksdb/}}
Third-party language bindings
Third-party programming language bindings available for RocksDB include:
{{Div col |colwidth=25em}}
- C
- C#{{cite web |url=https://github.com/warrenfalk/rocksdb-sharp |title=warrenfalk/rocksdb-sharp |website=GitHub |date=28 September 2021}}
- Chicken Scheme{{Cite web|url=http://wiki.call-cc.org/eggref/5/rocksdb|title=RocksDB bindings for CHICKEN Scheme 5|website=wiki.call-cc.org | access-date=2024-07-13}}
- D{{cite web |url=https://github.com/b1naryth1ef/rocksdb |title=b1naryth1ef/rocksdb |website=GitHub |date=22 October 2019}}
- Elixir{{cite web |url=https://github.com/urbint/rox |title=urbint/rox |website=GitHub |date=September 2021}}
- Erlang{{cite web |url=https://github.com/leo-project/erocksdb |title=leo-project/erocksdb |website=GitHub |date=September 2021}}{{Cite web|url=https://gitlab.com/barrel-db/erlang-rocksdb|title=barrel-db / erlang-rocksdb · GitLab|website=GitLab}}
- Go{{cite web |url=https://github.com/tecbot/gorocksdb |title=tecbot/gorocksdb |website=GitHub |date=29 October 2021}}
- Haskell{{Cite web|url=https://hackage.haskell.org/package/rocksdb-haskell|title=rocksdb-haskell|website=Hackage}}
- Java{{cite web |url=https://github.com/facebook/rocksdb/wiki/RocksJava-Basics |title=RocksJava |website=GitHub}}{{cite web |url=https://github.com/marykdb/rocksdb-android |title=RocksDB-Android |website=GitHub}}
- Kotlin{{cite web |url=https://github.com/marykdb/rocksdb-multiplatform |title=RocksDB Kotlin Multiplatform |website=GitHub}}
- Node.js{{cite web |url=https://npmjs.org/package/rocksdb |title=rocksdb|date=25 March 2022 }}
- Nim{{cite web |url=https://github.com/status-im/nim-rocksdb |title=rocksdb|website=GitHub }}
- Objective-C, and Swift{{cite web |url=https://github.com/iabudiab/ObjectiveRocks |title=iabudiab/ObjectiveRocks |website=GitHub |date=2 August 2021}}
- OCaml{{cite web |url=https://github.com/ahrefs/ocaml-ahrocksdb |title=OCaml bindings for RocksDB |website=GitHub |date=8 October 2021}}{{cite web |url=https://github.com/domsj/orocksdb |title=An OCaml RocksDb binding using ocaml-ctypes |website=GitHub |date=28 September 2020}}
- Perl{{Cite web|url=https://metacpan.org/pod/RocksDB|title=RocksDB|website=MetaCPAN}}
- PHP{{cite web |url=https://github.com/Photonios/rocksdb-php |title=Photonios/rocksdb-php |website=GitHub |date=11 August 2021}}
- Prolog{{Cite web|url=https://www.swi-prolog.org/pack/list?p=rocksdb|title="rocksdb" pack for SWI-Prolog|website=www.swi-prolog.org}}
- Python{{cite web |url=https://github.com/stephan-hof/pyrocksdb |title=stephan-hof/pyrocksdb |website=GitHub |date=27 October 2021}}
- Ruby{{Cite web|url=https://rubygems.org/gems/rocksdb-ruby|title=rocksdb-ruby | RubyGems.org | your community gem host|website=rubygems.org}}
- Rust{{cite web |url=https://github.com/rust-rocksdb/rust-rocksdb |title=rust-rocksdb/rust-rocksdb |website=GitHub |date=11 July 2024}}
{{Div col end}}
References
{{Reflist}}