Table Of Contents
Table of Contents |
---|
Needed Stories
LIFESPAN not to be supported in Casablanca.
Chameleon/Gallifrey need to handle all deployment stories (dockerization/runbook/etc), and adhere to junit coverage - amdocs to create these stories
Historical Meta-properties
Outlined in the following ppt: historyScenarios.pptx
Historical Work Breakdown
Historical Tracking Iterations Story Assignment.xlsx
mS Pros/Cons
Why are we breaking the functionality out into these granular levels (mainly for scalability, deployment/maintenance flexibility)
- Independent Development – All microservices can be easily developed based on their individual functionality
- Independent Deployment – Microservices can be individually deployed in any application
- Fault Isolation – Even if one service of the application does not work, the system still continues to function and the fault is easily detectable
- Mixed Technology Stack – Different languages and technologies can be used to build different services of the same application
- Granular Scaling – Individual components can scale as per need, there is no need to scale all components together
- Ease of Unit Testing - unit testing is easier to maintain as functionality per mS is isolated
Some disadvantages of splitting out the functionality into these granular microservices are:
- Full stack error log traceability
- Latency introduced
- Deployment more complicated
- e2e testing more complicated
- more points of failure
High Level Design of microservice flow
Resources - Client exposed endpoints
Gizmo - CRUD abstraction subsystem
Synapse - Data/request router, will handle the traffic proxying to various microservices based on its built in rules
Champ - General purpose graph database abstraction
Spike - publishes dmaap or kafka events and attempts to ensure order
Chameleon - Subsystem that processes spike events from the real-time graph updates and formats the events, entry point to gallifrey/time machine, and enforces/formats the requests into the format needed by Gallifrey/Time Machine
Gallifrey/Time Machine - Subsystem that makes & retrieves historical assertions
PUT/POST/PATCH/DELETE to real-time flow
Resources > Gizmo > Champ > Real-time DB Cluster
this triggers the historical storage flow
Champ > Spike > Chameleon > Gallifrey > Historical Champ > Historical DB Cluster
GET of real-time data
Resources > Gizmo > Synapse > Real-time Champ > Real-time DB Cluster
GET of historical data
Resources > Gizmo > Synapse > Chameleon > Gallifrey/Time Machine > Historical Champ > Historical DB Cluster
I feel Chameleon could be bypassed for this flow, treating chameleon's functionality as just the dmaap history processor, and synapse would call gallifrey/tm directly. This way if chameleon goes down, gallifrey/tm could still service historical GET requests.
Resources > Gizmo > Synapse > Chameleon > Gallifrey/Time Machine > Historical Champ > Historical DB Cluster
Resources API Spec
Resources will be updated to accept a timestamp or a network timestamp and will trickle down through gizmo to synapse then to chameleon.
If the timestamp was sent on a non-singular node call then we would return a message stating that this functionality is not supported.
Gallifrey/Time Machine API Spec:
Type | URI | Query Params | Description | Champ Interaction |
GET | relationship/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve a relationship by ID | Champ needs to handle accepting a relationship id and the timestamp to run a historical query on the graph db A subgraph strategy will be used in champ to filter on the relative timestamp provided. meta=[true/false] if true, payload retrieved will hold the metaproperties at t-k or t-t |
GET | relationship/<ID>/lifespan NOT in scope for Casablanca - decided on call 5/23/2018 | None | Retrieve all the timestamps for create, update, delete operations against this relationship | Champ would be called to retrieve the lifespan on the relationship with meta=true and t-k=lifespan to retrieve all the metaproperties on the relationship |
GET | entity/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve an entity by ID | Champ needs to handle accepting an entity id and the timestamp to run a historical query on the graph db retrieving the asserted state. Default not sending back metaproperties, if metaproperties are needed a parameter would need to be sent to champ. A subgraph strategy will be used in champ to filter on the relative timestamp provided. meta=[true/false] if true, payload retrieved will hold the metaproperties at t-k |
GET | entity/<ID>/lifespan NOT in scope for Casablanca - decided on call 5/23/2018 | None | Retrieve all the timestamps for create, update, delete operations against this entity | Champ would be called to retrieve the lifespan on the entity with meta=true and t-k=lifespan to retrieve all the metaproperties on the entity |
PUT | relationship/<ID> | actor=name of the system making the assertion changes-only=[true|false] if true, gallifrey/tm will actually determine what has changed between the PUT payload and the most recent set of assertions for the relationship. If false, the entire PUT body will be considered as a new set of assertions whether something has changed or not. create=[true|false] if true, Gallifrey/TM assumes that this is a create request, if false it assumes it is an update t- |
t=Timestamp that specifies |
a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that a relationship is to be created or updated (depending on the query parameters that are passed in). This API appends new assertions against the specified relationship. Generated in Gallifrey/TM t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM (why are we generating this here? if there is a maintenance issue our timings would be out of sync with when these took place in the real-time db) | create = false (changes-only true (execute diff)/false(assume everything changed)) When an assertion is being made without a network timestamp gallifrey/tm will call champ requesting the relationship with it's most current metaproperties. Gallifrey/TM would then adjust the metaproperties (of the updated properties) and would send the payload back to champ with a new current state and an updated previous state's metaproperties. Champ would override it's current metaproperty (for the updated/deleted properties) with the old and current metaproperties sent from Gallifrey/TM. For added properties, they would be added directly with the metaproperties sent from Gallifrey/TM. When an assertion is being made with a network timestamp gallifrey/tm will call champ requesting the relationship with all of its metaproperties. Gallifrey/TM would then insert the new assertion where appropriate (and adjust neighboring metaproperties) and send the modified payload back to champ for a replace. create = true POST - this would be a new create and Gallifrey/TM would pass the metaproperties on each of it's property values and on the relationship itself |
PUT | entity/<ID> | actor=name of the system making the assertion changes-only=[true|false] if true, gallifrey/tm will actually determine what has changed between the PUT payload and the most recent set of assertions for the entity. If false, the entire PUT body will be considered as a new set of assertions whether something has changed or not. create=[true|false] if true, Gallifrey/TM assumes that this is a create request, if false it assumes it is an update t- |
t=Timestamp that specifies |
a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that an entity is to be created or updated (depending on the query parameters that are passed in). This API appends new assertions against the specified |
entity. Generated in Gallifrey/TM t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM | create = false (changes-only true (execute diff)/false(assume everything changed)) When an assertion is being made without a network timestamp gallifrey/tm will call champ requesting the entity with it's most current metaproperties. Gallifrey/TM would then adjust the metaproperties (of the updated properties) and would send the payload back to champ with a new current state and an updated previous state's metaproperties. Champ would override it's current metaproperty (for the updated/deleted properties) with the old and current metaproperties sent from Gallifrey/TM. For added properties, they would be added directly with the metaproperties sent from Gallifrey/TM. When an assertion is being made with a network timestamp gallifrey/tm will call champ requesting the entity with all of its metaproperties. Gallifrey/TM would then insert the new assertion where appropriate (and adjust neighboring metaproperties) and send the modified payload back to champ for a replace. create = true POST - this would be a new create and Gallifrey/TM would pass the metaproperties on each of it's property values and on the entity itself | ||
DELETE | relationship/<ID> | actor=name of the system making the assertion t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that a relationship has been deleted. Generated in Gallifrey/TM t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey |
/TM | Gallifrey/TM would request the latest relationship from champ and would set all of it's properties dbEndTimes to t-k along with the dbEndTime on the relationship itself. | ||
DELETE | entity/<ID> | actor=name of the system making the assertion t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Asserts that an entity has been deleted. Generated in Gallifrey/TM t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey |
/TM | Gallifrey/TM would request the latest entity from champ and would set all of it's properties dbEndTimes to t-k along with the dbEndTime on the entity itself. |
Chameleon API Spec
Type | URI | Query Params | Description |
GET | relationship/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve a relationship by ID |
GET | entity/<ID> | t-k=Timestamp that specifies knowledge ie: when we received the assertion in Gallifrey/TM t-t=Timestamp that specifies a time assertion made by the client for when the change (update/add/delete) occurred in the network | Retrieve an entity by ID |
Between Chameleon->Gallifrey/TM, the following calls are made:
...
GET relationship/<ID>? t-k=<timestamp>
Need Chameleon updated with this:
GET entity/<ID>?t-t=<timestamp>
GET relationship/<ID>? t-t=<timestamp>
Champ API Spec
Every entity/relationship/property must have a metaproperty or set of meta properties, this will be enforced in champ.
POST an entity- For History provide meta-properties in the payload
Method=POST
https://<host>:9522/services/champ-service/v1/objects
Retrieve an entity across its lifespan
Method= GET, As default,meta=false
https://<host>:9522/services/champ-service/v1/objects/<key>
https://<host>:9522/services/champ-service/v1/objects/<key>&meta=true
(called when updating with network time)
https://<host>:9522/services/champ-service/v1/objects/<key>&meta=latest
(called when updating with only db time)
Retrieve an object for a given database timestamp
Method= GET, As default,meta=false
https://<host>:9522/services/champ-service/v1/objects/<key>?t-k=t1
https://<host>:9522/services/champ-service/v1/objects/<key>?t-k=t1&meta=true
Retrieve an object for a given network timestamp
Method= GET, As default,meta=false
https://<host>:9522/services/champ-service/v1/objects/<key>?t-t=t1
https://<host>:9522/services/champ-service/v1/objects/<key>?t-t=t1&meta=true
UPDATE an object
Method=PUT - default=[patch=false] which means full replace
https://<host>:9522/services/champ-service/v1/objects/<key>
https://<host>:9522/services/champ-service/v1/objects/<key>?patch=true
DELETE an object - DELETE
Method=PUT - default=[patch=false] which means full replace
https://<host>:9522/services/champ-service/v1/objects/<key>
https://<host>:9522/services/champ-service/v1/objects/<key>?patch=true
POST a relationship - For History provide meta-properties in the payload
Method: POST
URL: https://<host>:9522/services/champ-service/v1/relationships
Retrieve a relationship across its lifespan
Method= GET, As default,meta=false
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>&meta=true
Retrieve a relationship for a given database timestamp
Method= GET, As default,meta=false
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-k=t1
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-k=t1&meta=true
Retrieve a relationship for a given network timestamp
Method= GET, As default,meta=false
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-t=t1
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?t-t=t1&meta=true
UPDATE an object
Method=PUT - default=[patch=false] which means full replace
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>?patch=true
DELETE an object - DELETE
Method=PUT - default=[patch=false] which means full replace
URL: https://<host>:9522/services/champ-service/v1/relationships/<key>
Historical Tracking Metaproperties
dbStartTime - when an entity/property value/relationship was added to db
dbEndTime - when and entity or relationship was deleted from the db or a property's value was asserted to another state
ntStartTime - asserted by the client as to when the change took place in the network
ntEndTime - set when an assertion provided by the client make's the current state no longer true
startSOT - the source of truth that made the assertion
endSOT- the source of truth that made an assertion to make the current state no longer true
Schema
Separate db edge rules file: with all relationships many to many except parent child which could be one to many
Separate schema file: no properties on vertices are unique except for aai-uuid
Open question: what happens in the event that db edge rules change etc. (migrations)
When To Record T-K value (Live DB vs Historical DB)
Below is an explanation of the thoughts around when to record the t-k value, examples explaining outcome of storing at live vs historical DB and their pros/cons.
View file | ||||
---|---|---|---|---|
|
GUI Mocks
New integrated functionality (updates for history)
...