Very large database
{{Short description|Database that contains a very large amount of data}}
{{About|databases which are very large|the VLDB conference|International Conference on Very Large Data Bases}}
{{Use dmy dates|date=October 2018}}
A very large database, (originally written very large data base) or VLDB, is a database that contains a very large amount of data, so much that it can require specialized architectural, management, processing and maintenance methodologies.
Definition
The vague adjectives of very and large allow for a broad and subjective interpretation, but attempts at defining a metric and threshold have been made. Early metrics were the size of the database in a canonical form via database normalization or the time for a full database operation like a backup. Technology improvements have continually changed what is considered very large.
One definition has suggested that a database has become a VLDB when it is "too large to be maintained within the window of opportunity… the time when the database is quiet".
Sizes of a VLDB database
There is no absolute amount of data that can be cited. For example, one cannot say that any database with more than 1 TB of data is considered a VLDB. This absolute amount of data has varied over time as computer processing, storage and backup methods have become better able to handle larger amounts of data. That said, VLDB issues may start to appear when 1 TB is approached, and are more than likely to have appeared as 30 TB or so is exceeded.
VLDB challenges
=Configuration=
=Administration=
The complexities of managing a VLDB can increase exponentially for the database administrator as database size increases.
=Availability and maintenance=
When dealing with VLDB operations relating to maintenance and recovery such as database reorganizations and file copies which were quite practical on a non-VLDB take very significant amounts of time and resources for a VLDB database. In particular it typically infeasible to meet a typical recovery time objective (RTO), the maximum expected time a database is expected to be unavailable due to interruption, by methods which involve copying files from disk or other storage archives. To overcome these issues techniques such as clustering, cloned/replicated/standby databases, file-snapshots, storage snapshots or a backup manager may help achieve the RTO and availability, although individual methods may have limitations, caveats, license, and infrastructure requirements while some may risk data loss and not meet the recovery point objective (RPO). For many systems only geographically remote solutions may be acceptable.
==Backup and recovery==
=Performance=
Given the same infrastructure there may typically be a decrease in performance, that is increase in response time as database size increases. Some accesses will simply have more data to process (scan) which will take proportionally longer (linear time); while the indexes used to access data may grow slightly in height requiring perhaps an extra storage access to reach the data (sub-linear time). Other effects can be caching becoming less efficient because proportionally less data can be cached and while some indexes such as the B+ automatically sustain well with growth others such as a hash table may need to be rebuilt.
Should an increase in database size cause the number of accessors of the database to increase then more server and network resources may be consumed, and the risk of contention will increase. Some solutions to regaining performance include partitioning, clustering, possibly with sharding, or use of a database machine.{{rp|390}}
==Partitioning==
Partitioning may be able assist the performance of bulk operations on a VLDB including backup and recovery., bulk movements due to information lifecycle management (ILM),{{rp|3}}{{rp|105–118}} reducing contention{{rp|327–329}} as well as allowing optimization of some query processing.{{rp|215–230}}
=Storage=
In order to satisfy needs of a VLDB the database storage needs to have low access latency and contention, high throughput, and high availability.
=Server resources=
Relationship to big data
VLDB is not the same as big data, but the storage aspect of big data may involve a VLDB database. That said some of the storage solutions supporting big data were designed from the start to support large volumes of data, so database administrators may not encounter VLDB issues that older versions of traditional RDBMS's might encounter.
See also
References
{{reflist|2|refs=
|id=18 Very Large Databases (VLDB)|website=oracle|access-date=3 October 2018|url-access=limited}}
|title=Oracle Database Release 18 - VLDB and Partitioning Guide|id=1 Introduction to Very Large Databases|access-date=3 October 2018|website=Oracle|url-access=limited
|url-status=live|archive-url=https://web.archive.org/web/20181003205734/https://docs.oracle.com/en/database/oracle/oracle-database/18/vldbg/partition-intro.html|archive-date=3 October 2018|df=dmy-all}}
|url-status=live|archive-url=https://web.archive.org/web/20180704224849/https://www.techopedia.com/definition/14731/very-large-database-vldb|archive-date=4 July 2018|df=dmy-all}}
|url=http://www.redbrick.com/rbs-g/whitepapers/tenreq_wp.html
|url-status=dead|archive-url=https://web.archive.org/web/19971010114605/http://www.redbrick.com/rbs-g/whitepapers/tenreq_wp.html|archive-date=10 October 1997|df=dmy-all}}
|archive-url=https://web.archive.org/web/20181006114729/https://mwidlake.wordpress.com/2009/09/18/what-is-a-vldb/|archive-date=6 October 2018|df=dmy-all}}
|access-date=10 October 2018|first=Tim|last=Chien|url-status=live|website=Oracle technetwork
|archive-url=https://web.archive.org/web/20180907091910/https://www.oracle.com/technetwork/database/availability/rman-fra-snapshot-322251.html|archive-date=7 September 2018
|url-access=limited|df=dmy-all}}
|title=Using a split mirror as a backup image|website=IBM Knowledge Center|access-date=10 October 2018|url-access=limited
|url-status=live|archive-url=https://archive.today/20180109160158/https://www.ibm.com/support/knowledgecenter/en/SSEPGG_9.5.0/com.ibm.db2.luw.admin.ha.doc/doc/t0006423.html|archive-date=9 January 2018|df=dmy-all}}
|title=Understanding B+tree Indexes and how they Impact Performance|first=Ovais|last=Tariq|website=ovaistariq.net
|url-status=live|archive-url=https://web.archive.org/web/20180207203602/http://www.ovaistariq.net/733/understanding-btree-indexes-and-how-they-impact-performance/
|archive-date=7 February 2018|df=dmy-all|access-date=10 October 2018|date=2011-07-14}}
|url-status=live|archive-url=https://web.archive.org/web/20160704205410/http://www.pcmag.com/encyclopedia/term/40879/database-machine|archive-date=4 July 2016|df=dmy-all
|access-date=10 October 2018}}
|url-status=live|archive-url=https://web.archive.org/web/20170630223240/http://www.dba-oracle.com/t_backup_vldb.htm|archive-date=30 June 2017|df=dmy-all}}
|title=Database Backup and Recovery Best Practices|first1=Ali Navid|last1=Akhtar|first2=Jeff|last2=Buchholtz|first3=Michael|last3=Ryan|first4=Kumar|last4= Setty|date=2012|access-date=12 October 2012|url-status=live|archive-url=https://web.archive.org/web/20180629131442/https://www.isaca.org/Journal/archives/2012/Volume-1/Pages/Database-Backup-and-Recovery-Best-Practices.aspx|archive-date=29 June 2018|df=dmy-all}}
|title=Chapter 1 High Availability and Scalability|url-status=live|df=dmy-all|website=dev.mysql
|archive-url=https://web.archive.org/web/20161215030829/https://dev.mysql.com/doc/mysql-ha-scalability/en/ha-overview.html|archive-date=15 December 2016}}
|url=https://indico.cern.ch/event/697301/attachments/1598206/2532649/Partitioning_guide_v18.pdf|publisher=Oracle|website=Cern|location=40-S2-C01 - Salle Curie (CERN)|date=8 Feb 2018|access-date=12 October 2018
|archive-url=https://web.archive.org/web/20181012172456/https://indico.cern.ch/event/697301/attachments/1598206/2532649/Partitioning_guide_v18.pdf
|archive-date=12 October 2018|url-status=live|df=dmy-all}}
|title=Oracle Partitioning in Oracle Database 12c Release 2 Extreme Data Management and Performance for every System|date=March 2017|access-date=17 October 2018|website=Oracle
|archive-url=https://web.archive.org/web/20171215074909/https://www.oracle.com/technetwork/database/options/partitioning/partitioning-wp-12c-1896137.pdf
|url-access=limited|url-status=live|archive-date=15 December 2017|df=dmy-all}}
{{cite conference|conference=7th International Conference on Cloud Computing and Services
|title=High Availability and Performance of Database in the Cloud - Traditional Master-slave Replication versus Modern Cluster-based Solutions
|url=https://www.researchgate.net/publication/317299391
|first=Raju|last=Shrestha|doi=10.5220/0006294604130420|isbn=978-989-758-243-1|date=2017|volume=1: CLOSER|publisher= SCITEPRESS – Science and Technology Publications, Lda
|archive-url=https://web.archive.org/web/20181017152557/https://www.researchgate.net/publication/317299391_High_Availability_and_Performance_of_Database_in_the_Cloud_-_Traditional_Master-slave_Replication_versus_Modern_Cluster-based_Solutions|archive-date=17 October 2018|url-status=live|df=dmy-all|doi-access=free|hdl=10642/6140|hdl-access=free}}
|url=https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/hardware/|website=Microsoft IT Pro Center
|first1=Phil|last1=Steel|first2=Liza|last2=Poggemeyer|first3=Corey|last3=Plett
|archive-url=https://web.archive.org/web/20181017175544/https://docs.microsoft.com/en-us/windows-server/administration/performance-tuning/hardware/
|archive-date=17 October 2018|url-status=live|df=dmy-all}}
|first1=Charlotte|last1=Brooks|first2=Clem|last2=Leung|first3=Aslam|last3=Mirza|first4=Curtis|last4=Neal|first5=Yin Lei|last5=Qiu|first6=John|last6=Sing|first7=Francis TH|last7=Wong|first8=Ian R|last8=Wright|isbn=978-0738489728
|publisher=IBM Redbooks|date=March 2007}}
{{cite web|title=Cluster design considerations|access-date=17 October 2017|website=Crouchbase
|url=https://developer.couchbase.com/documentation/server/3.x/admin/Concepts/bp-clusterDesign.html
|archive-url=https://web.archive.org/web/20181017195247/https://developer.couchbase.com/documentation/server/3.x/admin/Concepts/bp-clusterDesign.html
|url-status=live|archive-date=17 October 2018|df=dmy-all}}
{{cite web|title=Cross Datacenter Replication (XDCR)|access-date=17 October 2017|website=Crouchbase
|url=https://developer.couchbase.com/documentation/server/3.x/admin/XDCR/xdcr-intro.html
|archive-url=https://web.archive.org/web/20181017195516/https://developer.couchbase.com/documentation/server/3.x/admin/XDCR/xdcr-intro.html
|url-status=live|archive-date=17 October 2018|df=dmy-all}}
|title=A performance comparison of SQL and NoSQL databases|date=2013|first1=Yishan|last1=Li|first2=Sathiamoorthy|last2=Manoharan|page=15
|doi=10.1109/PACRIM.2013.6625441|isbn=978-1-4799-1501-9|publisher=IEEE}}
}}
{{Database}}
{{DEFAULTSORT:Very Large Database}}