The major distinction between List and Object is that one, the List type uses a single file as backend with each record placed in one line.
Where as the Object type keeps each record in a separate file. The object type is/will keep track of these records with an index file, and also optionally with a strtree for fast in memory access.
How the List type works:
The user creates a struct to his/her needs
The database inherits that struct via an Interface
The database converts that struct to include the map id using jsonListStruct
jsonListStruct struct is used to Marshal the data in to a json file
What is planed so far:
Create a cache system with a user defined limit that keeps the most asked for items in a Tuple, as of now all data is being imported in to a Tuple.
Create the vcs (git) backup store for the json backend
Distribute with git push and pull
Keep an open line of communication with glob between nodes
git is not the fastest distribution system
All data is kept in plain text/json
Backups are incremental, distributed and under version control.
Distributed system. without the need for the master and slave concept.
Every node is independent of the overall system, and can function even if all others fail.
Database can get very big, bigger then what the memory can hold, and we will read less active data with io.Reader and bufio
To get a proper distributed system you need a merge strategy for your git repositories. How would you solve conflicts?
As with everything I am open to suggestions, meanwhile I am planing following:
The strategy will differ by List or Object type database.
For example the List type database will not have any update only insert and delete.
The insert operation will easily be resolved by git, when nodes do their pull, save, add, commit and push cycle.
When it comes to the delete operation, every node will before deleting a record, add lock on the file it plans to modify. Each node will be able to lock a file at pull time.
This means other nodes can still modify and commit all other files, just not the one locked by a node.
The lock will have a time limit as it does only takes a short time to perform a full cycle of pull, save, add, commit and push with git.