karbor/doc/source/api/bank.md

Bank basics

*** ❗ This is still a work in progress ***

This document will describe the layout and algorithms used by smaug using the bank interface.

Abstract

Since Smaug want's to be able to store metadata in many locations (swift, mongodb, etc.) we defined a simplified object store interface that we believe most backends will be able to support without much work.

But the simplified interface doesn't describe how Smaug will do it's higher level operations and how the higher level logic will be layed out in the object store. This is why we need higher level logic defined explicitly so that later we could use higher level bank functions knowing they are correct, safe and atomic.

Layout

Checkpoint directory

/checkpoints/<checkpoint_id>/index.json

Example content

{
    "trigger": {},
    "started_at": "11 OCT 2237",
    "status": "in progress",
    "plan": {},
    "plugins": {
        "plugin_1": {
            "name": "cinder volume",
            "version": "1.2.3",
        }
    }
}

Protection definition directory

/checkpoints/<Checkpoint_id>/<protection_definition_id>/index.json

Example content

{
    "plugin": "plugin_1",
    "dependencies": {
        "vol1": "<other_protection_defintion_id>"
    }
}

Protection definition plugin data directory

/checkpoints/<checkpoint_id>/<protection_defintion_id>/plugin_data/*

Checkpoint Creation Process

Create new Checkpoint with id ;

1. Acquire checkpoint lock

• action acquire_lock
• id: <CHECKPOINT-ID>

2. Create checkpoint pointer

• action: write_object
• path: /indices/unfinished_checkpoints/<CHECKPOINT-ID>,
• buffer: <CHECKPOINT-ID>

3. Create checkpoint

• action: write_object
• path: /checkpoints/<CHECKPOINT-ID>/index.json,
• buffer:

  {
      "smaug_version": "1.0.0",
      "status": "in_progress",
      "plugins": {}
  }
  

4. Run plugins
5. Checkpoint finished but indices not yet created

• action: write_object
• path: /checkpoints/<CHECKPOINT-ID>/index.json,
• buffer:

  {
      "smaug_version": "1.0.0",
      "status": "creating_indices",
      "plugins": {}
  }
  

6. Create index 'plan' (Example, there could be any number of indexes)

• action: write_object
• path: /indices/by_plan/<PLAN-ID>/<CHECKPOINT-ID>
• buffer: <CHECKPOINT-ID>

7. Remove checkpoint pointer

• action: delete_object
• path: /indices/unfinished_checkpoints/<CHECKPOINT-ID>

8. Release checkpoint lock

• action: release_lock
• id: <CHECKPOINT-ID>

Delete Checkpoint

1. Acquire checkpoint lock

• action acquire_lock
• id: <CHECKPOINT-ID>

2. Create checkpoint pointer

• action: write_object
• path: /indices/unfinished_checkpoints/<CHECKPOINT-ID>,
• buffer: <CHECKPOINT-ID>

3. Mark transaction as being deleted

• action: write_object
• path: /checkpoints/<CHECKPOINT-ID>/index.json,
• buffer:

  {
      "smaug_version": "1.0.0",
      "status": "deleting",
      "plugins": {}
  }
  

4. Remove indices

• Remove index 'plan' (Example, there could be any number of indexes)
  • action: delete_object
  • path: /indices/by_plan/<PLAN-ID>/<CHECKPOINT-ID>

6. Run plugins
7. Delete checkpoint file

• action: delete_object
• path: /checkpoints/<CHECKPOINT-ID>/index.json,

8. Remove checkpoints pointer

• action: delete_object
• path: /indices/unfinished_checkpoints/<CHECKPOINT-ID>

9. Release checkpoint lock

• action: release_lock
• id: <CHECKPOINT-ID>