Manually regenerated since Makefile webpublish target seems broken

Author: danshearer

README

@@ -0,0 +1 @@
+This website automatically generated on Fri 10 Jul 19:59:29 BST 2020 by https://github.com/LumoSQL/doc/www/Makefile

README.md

@@ -22,6 +22,7 @@ LumoSQL emphasises benchmarking, code reuse and modern database implementation.
 
 * [Quick Start](./lumo-quickstart.md)
 * [LumoSQL Project Aims](./lumo-project-aims.md)
+* [Code of Conduct](./lumo-code-of-conduct.md)
 * Creating a LumoSQL Ecosystem
     + [The LumoSQL Landscape](./lumo-landscape.md)
     + [Codebases relevant to LumoSQL](./lumo-relevant-codebases.md)

images/lumo-architecture-intro.jpg renamed to lumo-architecture-intro.jpg

@@ -0,0 +1,152 @@
+# LumoSQL Code of Conduct
+
+Version 1.1 – Updated May 1st, 2020
+
+*Adapted from version 3.1 of the 
+[Mozilla Participation Guidelines](https://www.mozilla.org/en-US/about/governance/policies/participation/)
+and released under the [Creative Commons Attribution-ShareAlike 4.0 International](https://creativecommons.org/licenses/by-sa/4.0/) license.*
+
+The heart of LumoSQL is people. We put people first and do our best to recognize, appreciate and respect the diversity of our global contributors. The LumoSQL Project welcomes contributions from everyone who shares our goals and wants to contribute in a healthy and constructive manner within our community. As such, we have adopted this code of conduct and require all those who participate to agree and adhere to these Community Participation Guidelines in order to help us create a safe and positive community experience for all.
+
+These guidelines aim to support a community where all people should feel safe to participate, introduce new ideas and inspire others, regardless of:
+
+- Background
+    Family status
+    Gender
+    Gender identity or expression
+    Marital status
+    Sex
+    Sexual orientation
+    Native language
+    Age
+    Ability
+    Race and/or ethnicity
+    Caste
+    National origin
+    Socioeconomic status
+    Religion
+    Geographic location
+    Any other dimension of diversity
+
+Openness, collaboration and participation are core aspects of our work. We gain strength from diversity and actively seek participation from those who enhance it. These guidelines exist to enable diverse individuals and groups to interact and collaborate to mutual advantage. This document outlines both expected and prohibited behavior.
+
+# This Code of Conduct Applies A Lot of the Time
+
+These guidelines outline our behavior expectations as members of the LumoSQL community in all LumoSQL activities, both offline and online. Your participation is contingent upon following these guidelines in all LumoSQL activities, including but not limited to:
+
+    * Working with other LumoSQL community participants virtually or co-located
+    * Representing LumoSQL at public events
+    * Representing LumoSQL in social media
+    * Participating in LumoSQL-related forums, mailing lists, wikis, websites, chat channels, bugs, group or person-to-person meetings, and LumoSQL-related correspondence.
+
+# Expected Behaviour
+
+The following behaviors are expected of all LumoSQL community participants:
+
+## Be Respectful
+
+Value each other’s ideas, styles and viewpoints. We may not always agree, but disagreement is no excuse for poor manners. Be open to different possibilities and to being wrong. Be respectful in all interactions and communications, especially when debating the merits of different options. Be aware of your impact and how intense interactions may be affecting people. Be direct, constructive and positive. Take responsibility for your impact and your mistakes – if someone says they have been harmed through your words or actions, listen carefully, apologize sincerely, and correct the behavior going forward.
+
+## Be Direct but Professional
+
+We are likely to have some discussions about if and when criticism is respectful and when it’s not. We must be able to speak directly when we disagree and when we think we need to improve. We cannot withhold hard truths. Doing so respectfully is hard, doing so when others don’t seem to be listening is harder, and hearing such comments when one is the recipient can be even harder still. We need to be honest and direct, as well as respectful.
+
+## Be Inclusive
+
+Seek diverse perspectives. Diversity of views and of people on teams powers innovation, even if it is not always comfortable. Encourage all voices. Help new perspectives be heard and listen actively. If you find yourself dominating a discussion, it is especially important to step back and encourage other voices to join in. Be aware of how much time is taken up by dominant members of the group. Provide alternative ways to contribute or participate when possible.
+
+Be inclusive of everyone in an interaction, respecting and facilitating people’s participation whether they are:
+
+    * Remote (on video or phone)
+    * Not native language speakers
+    * Coming from a different culture
+    * Using pronouns other than “he” or “she”
+    * Living in a different time zone
+    * Facing other challenges to participate
+
+Think about how you might facilitate alternative ways to contribute or participate. If you find yourself dominating a discussion, step back. Make way for other voices and listen actively to them.
+
+## Understand Different Perspectives
+
+Our goal should not be to “win” every disagreement or argument. A more productive goal is to be open to ideas that make our own ideas better. Strive to be an example for inclusive thinking. “Winning” is when different perspectives make our work richer and stronger.
+
+## Appreciate and Accommodate Our Similarities and Differences
+
+LumoSQL community participants come from many cultures and backgrounds. Cultural differences can encompass everything from official religious observances to personal habits to clothing. Be respectful of people with different cultural practices, attitudes and beliefs. Work to eliminate your own biases, prejudices and discriminatory practices. Think of others’ needs from their point of view. Use preferred titles (including pronouns) and the appropriate tone of voice. Respect people’s right to privacy and confidentiality. Be open to learning from and educating others as well as educating yourself; it is unrealistic to expect LumoSQL community participants to know the cultural practices of every ethnic and cultural group, but everyone needs to recognize one’s native culture is only part of positive interactions.
+
+## Lead by Example
+
+By matching your actions with your words, you become a person others want to follow. Your actions influence others to behave and respond in ways that are valuable and appropriate for our organizational outcomes. Design your community and your work for inclusion. Hold yourself and others accountable for inclusive behaviors. Make decisions based on the highest good for LumoSQL’s mission.
+
+# Behavior That Will Not Be Tolerated
+
+The following behaviors are considered to be unacceptable under these guidelines.
+
+## Violence and Threats of Violence
+
+Violence and threats of violence are not acceptable - online or offline. This includes incitement of violence toward any individual, including encouraging a person to commit self-harm. This also includes posting or threatening to post other people’s personally identifying information (“doxxing”) online.
+
+## Personal Attacks
+
+Conflicts will inevitably arise, but frustration should never turn into a personal attack. It is not okay to insult, demean or belittle others. Attacking someone for their opinions, beliefs and ideas is not acceptable. It is important to speak directly when we disagree and when we think we need to improve, but such discussions must be conducted respectfully and professionally, remaining focused on the issue at hand.
+
+## Derogatory Language
+
+Hurtful or harmful language related to:
+
+    * Background
+    * Family status
+    * Gender
+    * Gender identity or expression
+    * Marital status
+    * Sex
+    * Sexual orientation
+    * Native language
+    * Age
+    * Ability
+    * Race and/or ethnicity
+    * Caste
+    * National origin
+    * Socioeconomic status
+    * Religion
+    * Geographic location
+    * Other attributes
+
+is not acceptable. This includes deliberately referring to someone by a gender that they do not identify with, and/or questioning the legitimacy of an individual’s gender identity. If you’re unsure if a word is derogatory, don’t use it. This also includes repeated subtle and/or indirect discrimination; when asked to stop, stop the behavior in question.
+
+## Unwelcome Sexual Attention or Physical Contact
+
+Unwelcome sexual attention or unwelcome physical contact is not acceptable. This includes sexualized comments, jokes or imagery in interactions, communications or presentation materials, as well as inappropriate touching, groping, or sexual advances. This includes touching a person without permission, including sensitive areas such as their hair, pregnant stomach, mobility device (wheelchair, scooter, etc) or tattoos. This also includes physically blocking or intimidating another person. Physical contact or simulated physical contact (such as emojis like “kiss”) without affirmative consent is not acceptable. This includes sharing or distribution of sexualized images or text.
+
+## Disruptive Behavior
+
+Sustained disruption of events, forums, or meetings, including talks and presentations, will not be tolerated. This includes:
+
+    * ‘Talking over’ or ‘heckling’ speakers.
+    * Drinking alcohol to excess or using recreational drugs to excess, or pushing others to do so.
+    * Making derogatory comments about those who abstain from alcohol or other substances, pushing people to drink, talking about their abstinence or preferences to others, or pressuring them to drink - physically or through jeering.
+    * Otherwise influencing crowd actions that cause hostility in the session.
+
+## Influencing Unacceptable Behavior
+
+We will treat influencing or leading such activities the same way we treat the activities themselves, and thus the same consequences apply.
+
+# Consequences of Unacceptable Behavior
+
+Bad behavior from any LumoSQL community participant, including those with decision-making authority, will not be tolerated. Intentional efforts to exclude people (except as part of a consequence of the guidelines or other official action) from LumoSQL activities are not acceptable and will be dealt with appropriately.
+
+Reports of harassment/discrimination will be promptly and thoroughly investigated by the people responsible for the safety of the space, event or activity. Appropriate measures will be taken to address the situation.
+
+Anyone asked to stop unacceptable behavior is expected to comply immediately. Violation of these guidelines can result in you being ask to leave an event or online space, either temporarily or for the duration of the event, or being banned from participation in spaces, or future events and activities in perpetuity.
+
+In addition, any participants who abuse the reporting process will be considered to be in violation of these guidelines and subject to the same consequences. False reporting, especially to retaliate or exclude, will not be accepted or tolerated.
+
+# Reporting
+
+If you believe you’re experiencing unacceptable behavior that will not be tolerated as outlined above please contact one of the [authors](https://github.com/LumoSQL/LumoSQL/blob/master/AUTHORS).
+
+After receiving a concise description of your situation, they will review and determine next steps. In addition to conducting any investigation, they can provide a range of resources, from a private consultation to other community resources. They will involve other colleagues or outside specialists, as needed to appropriately address each situation.
+
+Please also report to us if you observe a potentially dangerous situation, someone in distress, or violations of these guidelines, even if the situation is not happening to you.
+
+If you feel you have been unfairly accused of violating these guidelines, please follow the same reporting process.

images/lumo-architecture-lumosql-theoretical-future.svg renamed to lumo-architecture-lumosql-theoretical-future.svg

@@ -29,7 +29,7 @@ Table of Contents
 LumoSQL Documentation Standards
 ===============================
 
-This chapter covers how LumoSQL documentation should be written and maintained. 
+This chapter covers how LumoSQL documentation is written and maintained. 
 
 ![](./images/lumo-doc-standards-intro.jpg "Image from Wikimedia Commons, https://commons.wikimedia.org/wiki/File:Chinese_books_at_a_library.jpg")
 
@@ -157,7 +157,7 @@ This example is approximately from the top of this chapter:
 It's best to check syntax before pushing changes, which means rendering
 Markdown into HTML that is hopefully close to what Github produces. Here are three ways of doing that:
 
-* The Makefile and support files in bin/ uses Pandoc to render the GFM to HTML in /tmp .
+* The Makefile and support files in bin/ uses Pandoc to render the GFM to HTML in /tmp . Just type 'make 
 * The excellent [Editor.md](https://github.com/pandao/editor.md) does a great job of rendering,
 as can be seen at [The Online Installation](https://pandao.github.io/editor.md/en.html) . You can paste GFM into it and see it rendered, WYSIWYG-style. You can download the HTML for
 Editor.md and run it locally. (Editor.md is also an editor, and it adds its own features, but you don't need to use it for that.)

images/lumo-architecture-online-db-server-scale.svg renamed to lumo-architecture-online-db-server-scale.svg

@@ -45,7 +45,7 @@ Aims
   backends, frontends, encryption, networking and more. 
 
 * Devops contract: LumoSQL will reduce risk by making it possible to omit
-  compliation of many features, and will have stable ABIs ([Application Binary Interfaces](https://en.wikipedia.org/wiki/Application_binary_interface))so as to not break dynamically-linked applications.
+  compliation of unneeded features, and will have stable ABIs ([Application Binary Interfaces](https://en.wikipedia.org/wiki/Application_binary_interface)) so as to not break dynamically-linked applications.
 
 * Ecosystem creation: LumoSQL will offer consolidated contact, code curation, bug tracking,
   licensing, and community communications across all these features from
@@ -57,20 +57,20 @@ Short Term Goals
 ================
 
 * LumoSQL will have three canonical and initial backends: btree (the existing
-SQLite btree, ported to a new backend system); a test backend such as text or
-csv; and the LMDB backend. Control over these interfaces will be through the
+SQLite btree, ported to a new backend system); the LMDB backend; and the BDB
+backend. Control over these interfaces will be through the
 same user interface mechanisms as the rest of LumoSQL, and SQLite.
 
 * LumoSQL will improve SQLite quality and privacy compliance by introducing
 optional on-disk checksums for storage backends including to the original
-SQLite btree format.  This will give real-time row-level corruption detection.
+SQLite btree format.  This will give real-time row-level corruption detection
 
 * LumoSQL will improve SQLite quality and privacy compliance by introducing
-optional storage backends that are more crash-resistent, starting with LMDB
-followed by others.
+optional storage backends that are more crash-resistent than SQLite btree (such as LMDB)
+and more oriented towards complete recovery (such as BDB)
 
 * LumoSQL will improve SQLite integrity in persistent storage by introducing
-optional row-level checksums.
+optional row-level checksums
 
 * LumoSQL will provide the benefits of Open Source and an open project
 by continuing to accept and review contributions in an open way, using
@@ -96,22 +96,19 @@ tools
 
 * LumoSQL will ensure that new code remains optional by means of modularity at
 compiletime and also runtime. By illustration of modularity, at compiletime
-nearly all 30 million lines of the Linux kernel can be exclude giving just 200k
+nearly all 30 million lines of the Linux kernel can be excluded giving just 200k
 lines. Runtime modularity will be controlled through the same user interfaces 
 as the rest of LumoSQL.
 
-* LumoSQL will ensure that new code can all be active at once, eg
+* LumoSQL will ensure that new code may be active at once, eg
 multiple backends or frontends for conversion between/upgrading from one
-format or protocol to another. This is crucial to provide continuity and
+format or protocol to another. This is important to provide continuity and
 supported upgrade paths for users, for example, users who want to become
 privacy-compliant without disrupting their end users
 
 * Over time, LumoSQL will carefully consider the potential benefits of dropping
 some of the most ancient parts of SQLite when merging from upstream, provided
 it does not conflict with any of the other goals in this document. Eliminating 
 SQLite code can be done by a similar non-forking mechanism as used to keep in synch
-with the SQLite upstream.
-
-
-
+with the SQLite upstream. Patches will be offered to sqlite.org
 

images/lumo-architecture-online-db-server.svg renamed to lumo-architecture-online-db-server.svg

@@ -62,6 +62,11 @@ MVCC-compliant K-V stores, and still using Write-Ahead Logs.
 
 # Oracle BDB and Oracle BDB-SQL Codebases
 
+As of June 2020, Oracle announced that it has dropped support for the BDB port
+to SQLite. DB 18.1.32 is the last version to carry this, which is based on
+SQLite from 2017. This is the reference and basis for the BDB backend in
+LumoSQL.
+
 | Project | Last modified | Description   |
 | ------------- | ------------- | --------|
 | [Sleepycat/Oracle BDB](https://fossies.org/linux/misc/db-18.1.32.tar.gz) | current | The original ubiquitous Unix K-V store, disused in open source since Oracle's 2013 license change; the API template for most of the k-v btree stores around. Now includes many additional features including full MVCC transactions, networking and replication. This link is a mirror of code from download.oracle.com, which requires a login | 

images/lumo-architecture-sqlite-overview.svg renamed to lumo-architecture-sqlite-overview.svg

@@ -78,11 +78,47 @@ Analyser, DB Browser for SQLite, Magnet AXIOM and Oxygen Forensic Detective.)
 
 # List of Relevant Benchmarking and Test Knowledge
 
-Benchmarking is a big part of LumoSQL, to determine if changes are an improvement. The trouble is that SQLite and other top databases are not really benchmarked in realistic and consistent way, despite SQL server benchmarking using tools like TPC being an obsessive industry in itself, and there being myriad of testing tools released with SQLite, Postgresql, MariaDB etc. But in practical terms there is no way of comparing the most-used databases with each other, or even of being sure that the tests that do exist are in any way realistic, or even of simply reproducing results that other people have found. LumoSQL covers so many codebases and use cases that better SQL benchmarking is a project requirement. Benchmarking and testing overlap, which is addressed in the code and docs.
+Benchmarking is a big part of LumoSQL, to determine if changes are an
+improvement. The trouble is that SQLite and other top databases are not really
+benchmarked in realistic and consistent way, despite SQL server benchmarking
+using tools like TPC being an obsessive industry in itself, and there being
+myriad of testing tools released with SQLite, Postgresql, MariaDB etc. But in
+practical terms there is no way of comparing the most-used databases with each
+other, or even of being sure that the tests that do exist are in any way
+realistic, or even of simply reproducing results that other people have found.
+LumoSQL covers so many codebases and use cases that better SQL benchmarking is
+a project requirement. Benchmarking and testing overlap, which is addressed in
+the code and docs.
+
+The well-described [testing of SQLite](https://sqlite.org/testing.html)
+involves some open code, some closed code, and many ad hoc processes. Clearly
+the SQLite team have an internal culture of testing that has benefitted the
+world. However that is very different to reproducible testing, which is in turn
+very different to reproducible benchmarking, and that is even without
+considering whether the benchmarking is a reasonable approximation of actual
+use cases. As the development of LumoSQL has proceeded, it has become clear
+that the TCL testing harness shipped with SQLite code contains specific
+dependencies on the behaviour of the SQLite btree backend. While LumoSQL with
+the original btree backend aims to always pass these tests, differences such as
+locking behaviour, assumed key lengths, and even the number of database files a
+backend uses all mean that the SQLite TCL test suite is not generally useful.
+
+To highlight how poorly SQL benchmarking is done: there are virtually no test
+harnesses that cover encrypted databases and/or encrypted database connections,
+despite encryption being frequently required, and despite crypto implementation
+decisions making a very big difference in performance. Encryption and security are
+not the only ways a database impacts privacy, so privacy is a valid dimension for 
+database testing - and a fundamental goal for LumoSQL. Testing all databases in 
+the same way for privacy is challenging.
+
+SQL testing is also very difficult. As the Regression Testing paper below says:
+"A problem with testing SQL in DBMSs lies in the fact that the state of the
+database must be considered when deducing testing outcomes". SQL statements
+frequently change the state of the server during their execution, in different
+ways on different servers. This can change the behaviour or at least the
+performance of the next statement, and so on.
 
-The well-described [testing of SQLite](https://sqlite.org/testing.html) involves some open code, some closed code, and many ad hoc processes. Clearly the SQLite team have an internal culture of testing that has benefitted the world. However that is very different to reproducible testing, which is in turn very different to reproducible benchmarking, and that is even without considering whether the benchmarking is a reasonable approximation of actual use cases.
 
-To highlight how poorly SQL benchmarking is done: there are virtually no test harnesses that cover encrypted databases and/or encrypted database connections, despite encryption being frequently required, and despite crypto implementation decisions making a very big difference in performance.
 
 | Project | Last modified | Description | 
 | ------- | ------------- | ----------- |
@@ -92,6 +128,10 @@ To highlight how poorly SQL benchmarking is done: there are virtually no test ha
 | [Yahoo Cloud Serving Benchmark](https://github.com/brianfrankcooper/YCSB/)| current | Benchmarking tool for K-V stores and cloud-accessible databases |
 | [Example Android Storage Benchmark](https://github.com/greenrobot/android-database-performance) | 2018 | This code is an example of the very many Android benchmarking/testing tools. This needs further investigation |
 | [Sysbench](https://github.com/akopytov/sysbench) | current | A multithreaded generic benchmarking tool, with one well-supported use case being networked SQL servers, and [MySQL in particular](https://www.percona.com/blog/2019/04/25/creating-custom-sysbench-scripts/) |
+| [Regression Testing of SQL](https://www.diva-portal.org/smash/get/diva2:736996/FULLTEXT01.pdf)|n/a | 2014 paper "a framework for minimizing the required developer effort formanaging and running SQL regression tests" |
+| [Enhancing the Performance of SQLite](https://pdfs.semanticscholar.org/c2da/33304627649b599f80a5428354e116ba6201.pdf)| n/a | 2019 paper that does profiling and develops performance testing metrics for SQLite |
+| [SQLite Profiler and Tracer](https://github.com/microsoft/sqlite-tracer) | 2018 | Microsoft SQLite statement profiler and timer, helpful for comparing LumoSQL backends |
+| [SQLCipher Performance Optimisation](https://discuss.zetetic.net/t/sqlcipher-performance-optimization/14) | n/a | 2014 comments on the additional performance metric problems that come with SQLite encryption |
 
 
 # List of Just a Few SQLite Encryption Projects