Scalable, Practical Deterministic Database Systems

Overview

This work is funded by the NSF under grant IIS-1527118 which became IIS-1763797 when it was moved to UMD. The title of this NSF award is: "Scalable, Practical Deterministic Database Systems". The start date (of the original grant before it moved to UMD) was September 1, 2015 and the end date is August 31, 2019.

The database system was one of the first highly concurrent systems ever designed, and has served as a blueprint for the design of many subsequent concurrent computing systems. The decision to allow the system to process concurrent transactions nondeterministically has led to countless headaches from bugs (and debugging), security, replication, and general code complexity. The research funded by this grant is investigating the development of a fundamentally different design for concurrent execution of transactions in database systems that guarantees that the final state of the database is deterministically prescribed from the input to the system. Such databases have the potential to significantly improve the throughput, latency, fault tolerance, and general robustness of database systems.

Relevant papers published prior to this grant

• VLL: A Lock Manager Redesign for Main Memory Database Systems
  Kun Ren, Alexander Thomson, and Daniel J. Abadi. VLDB Journal 24(5): 681-705, October 2015. (bibtex)


• Rethinking serializable multiversion concurrency control
  Jose M. Faleiro and Daniel J. Abadi. PVLDB 8(11): 1190-1201, 2015. (bibtex)


• Lazy Evaluation of Transactions in Database Systems
  Jose Faleiro, Alexander Thomson, and Daniel J. Abadi. In Proceedings of SIGMOD, 2014. (bibtex)


• An Evaluation of the Advantages and Disadvantages of Deterministic Database Systems
  Kun Ren, Alexander Thomson, and Daniel J. Abadi. PVLDB 7(10): 821-832, 2014. (bibtex)


• Modularity and Scalability in Calvin
  Alexander Thomson and Daniel J. Abadi. IEEE Data Engineering Bulletin, 36(2): 48-55, 2013. (bibtex)


• Calvin: Fast Distributed Transactions for Partitioned Database Systems
  Alexander Thomson, Thaddeus Diamond, Shu-Chun Weng, Kun Ren, Philip Shao, and Daniel J. Abadi. In Proceedings of SIGMOD, 2012. (bibtex)

Publications supported by this grant

• SLOG: Serializable, Low-latency, Geo-replicated Transactions
  Kun Ren, Dennis Li, and Daniel J. Abadi. PVLDB 12(11), 2019. (bibtex)


• It's Time to Move on from Two Phase Commit
  Daniel J. Abadi. DBMS Musings. http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-from-two-phase.html, 2019. (bibtex)


• An Overview of Deterministic Database Systems
  Daniel J. Abadi and Jose M. Faleiro. CACM, 61(9), September 2018. (bibtex)


• Partitioned consensus and its impact on Spanner's latency
  Daniel J. Abadi. DBMS Musings. http://dbmsmusings.blogspot.com/2018/12/partitioned-consensus-and-its-impact-on.html, 2018. (bibtex)


• NewSQL database systems are failing to guarantee consistency, and I blame Spanner
  Daniel J. Abadi. DBMS Musings. http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-failing-to.html, 2018. (bibtex)


• High Performance Transactions via Early Write Visibility
  Jose M. Faleiro, Daniel J. Abadi, and Joseph M. Hellerstein. PVLDB 10(5): 613-624, 2017. (bibtex)


• Latch-free Synchronization in Database Systems: Silver Bullet or Fool's Gold?
  Jose M. Faleiro and Daniel J. Abadi. In Proceedings of CIDR, 2017. (bibtex)


• Automatic Generation of Normalized Relational Schemas from Nested Key-Value Data
  Michael DiScala and Daniel J. Abadi. In Proceedings of SIGMOD, 2016. (bibtex)


• Design Principles for Scaling Multi-core OLTP Under High Contention
  Kun Ren, Jose Faleiro, and Daniel J. Abadi. In Proceedings of SIGMOD, 2016. (bibtex)


• Low-Overhead Asynchronous Checkpointing in Main-Memory Database Systems
  Kun Ren, Thaddeus Diamond, Daniel J. Abadi, and Alexander Thomson. In Proceedings of SIGMOD, 2016. (bibtex)


• Scalable Pattern Matching Over Compressed Graphs via Dedensification
  Antonio Maccioni and Daniel J. Abadi. In Proceedings of KDD, 2016. (bibtex)


• LEOPARD: Lightweight Edge-Oriented Partitioning and Replication for Dynamic Graphs
  Jiewen Huang and Daniel J. Abadi. PVLDB 9(7): 540-551, 2016. (bibtex)

Code

• https://github.com/yaledb/calvin

People

PI

• Daniel Abadi

Post-doc:

• Kun Ren

Graduate Students:

• Alexander Thomson
• Jose Faleiro
• Jiewen Huang
• Antonio Maccioni

Undergraduate Students:

• Thaddeus Diamond
• Michael Hopkins
• Matthew Antoszyk
• Michael DiScala
• Anthony Tordillos