Jira | ||||||||
---|---|---|---|---|---|---|---|---|
|
Table of Contents |
---|
Overview
CPS temporal data query will allow allows the user to fetch data based on multiple filtering criteria. This data can be used to create graphs and help the by an analytical system to take decisions.
Filtering
...
Criteria
As it is time-series data, it can be filtered based on three main criteria:
Datetime
- Data after specified DateTime (To be supported in Istanbul Release)
- Data before specified
Datetime
- Data in last X hours
- Data after a particular DateTime Data before a particular DateTime
- Last X network datarecords
DataTypes in CPS Core
Dataspace & schema-
setset (To be supported in Istanbul Release)
- Dataspace & anchoranchor (To be supported in Istanbul Release)
- Dataspace & multiple anchors - To improve performance, if there is a need to fetch data for multiple anchors.
- Dataspace Payload
Payload (To be partially supported in Istanbul Release)
Based on a
subsection orfield in the payload. These criteria
doneed schema-set to be fixed to search in the same set of anchors. The details of the format are defined here.
We will implement only the basic APIs first and will add complex ones when required.
Other
...
Parameters
There are other parameters, listed below, that can have an impact on the response.
No | Name | Type | Default | Purpose | Example | ||
---|---|---|---|---|---|---|---|
1 | indexpageNumber | int | 0 | The query output can have many rows so it is important to limit the fetched data. To limit the number of records and to provide pagination, these two parameters can be used. pageNumber starts from "index" parameter represents the page number, starts with 0, and maxSize is the page size.0" pageLimit can't be greater than the maximum value configured in the application | |||
2 | pageLimit | 2 | maxSize | int | 1000 | ||
3 | pointInTime | DateTime | CurrentDateTime | Pagination does not work well if new data gets added which fulfils the search criteria. The user must provide this value to avoid this issue.Timestamp (ISO Format) | CurrentDateTime | When a user uses the APIs with pagination (by running several stateless requests), the result set can be impacted by new states stored while navigating. This can be a challenge to retrieve a consistent result set. This parameter must be used to avoid this issue. However, It will not provide a consistent result set if data, that meet search criteria, get deleted because of the cleanup triggered by the retention policy. | |
4 | sort | string | timestamp:desc | Data is by default returned sorted by the timestamp in descending order. It can be overridden and data can be sorted by multiple fields. | sort=timestamp:asc,anchor:desc |
Assumptions
- Temporal database stores full information in the payload ( CPS-192: Design data store for Temporal Service). It is possible that data stored for the different timestamps is the same, especially if the payload is used to filter data.
- Data in the response body will always contain the entire payload data.
- If the payload is passed as XPath, it is not expected to support all cps-path query and XPath functionalities.
Proposed APIs
...
Return data entries for an anchor after the specified Datetime, which matches the payload format.
...
Return data entries based on provided schema-set after the specified Datetime, which matches the payload format.
Approach 1 - GET
In the GET APIs, all the filtering parameters are passed as query parameter.
The payload can be passed as JSON or XPath. But, XPath is more concise and more readable than JSON when passed as a query parameter.
...
Payload Filter Format
Anchor | ||||
---|---|---|---|---|
|
The data can be filtered, using JSON format, on one field at any level, but it is important to provide the entire path from the root. The below JSON is used as an example to explain the filtering format
Code Block | ||
---|---|---|
| ||
{
"idNearRTRIC": "1",
"labels" : [ "cps", "ran"],
"nestedObject": {
"id": "4",
"attributes": {
"gNBDUId": 4.0,
"gNBDUName": "gnduserver4",
"gNBIdLength": 32.0
}
},
"GNBDUFunction": [
{
"idGNBDUFunction": "5",
"NRCellDU": [
{
"idNRCellDU": "13905",
"attributes": {
"nRPCI": 12.0,
"nRTAC": 310.0,
"cellLocalId": 13905.0
}
},
{
"idNRCellDU": "14655",
"attributes": {
"nRPCI": 12.0,
"nRTAC": 310.0,
"cellLocalId": 14655.0
}
}
],
"attributes": {
"gNBDUId": 5.0,
"gNBDUName": "gnduserver5",
"gNBIdLength": 32.0
}
}
]
} |
The filtering scenarios and the format are defined below:
No | Filter Condition | To be search | Format |
---|---|---|---|
1 | Top-level Field | idNearRTRIC | { "idNearRTRIC": "1"} |
2 | Nested Field | nestedObject → id | { "nestedObject" : {"attributes" : {"gNBDUId": 4.0 }} } |
3 | Value(s) in a collection | labels | { "labels": ["cps"] } |
4 | Field of an object inside a collection | GNBDUFunction → idGNBDUFunction | { "GNBDUFunction" : [ {"idGNBDUFunction" : 5 }] } |
Proposed APIs
The GET Endpoints contains information about the anchors and the rest of the filtering criteria are passed as query parameter. The below-mentioned query parameters are supported by all endpoints.
- after - timestamp ( ISO Format) to consider data created after the specified Timestamp
- simplePayloadFilter - to define payload filtering criteria
- pageNumber - to define the page number (required for pagination)
- pageLimit - to limit the response size
- pointInTime - to consider data changed or added before this time.
- sort - to define the order of data in the result set
The response format for endpoints is the same and defined here
By Dataspace and Anchor
URL: GET /dataspaces/{dataspace-name}/anchors/{anchor-name}/history
Example: GET /dataspaces/{dataspace-name}/anchors/{anchor-name}/history?after=2021-03-21T00:00:00-0:00&
...
simplePayloadFilter={"idNearRTRIC":"1"}&pageLimit=500
By Dataspace and Schema-set
...
URL:
GET /dataspaces/{dataspace-name}
...
/anchors/history?schemaSet={schemaset-name}
Example: GET /dataspaces/{dataspace-name}/anchors?schemaSet={schemaset-name}&after=2021-03-21T00:00:00-0:00
...
&simplePayloadFilter={"idNearRTRIC":"1"}
Response Anchor responseBody responseBody
responseBody | |
responseBody |
...
name | type | |
---|---|---|
nextRecordsLink | string | added only if there are remaining records to be fetched for the query. |
previousRecordsLink | string | added only if it is not the first set of records. |
records | list | contains one record for each timestamp that meets filtering criteria. It contains header information along with data. |
...
Code Block |
---|
{ "nextRecordsLink": "cps-temporal/api/v1/dataspaces/{dataspace-name}/anchors/{anchor-name}/history?after=2021-03-21T00:00:00-0:00&maxSizesimplePayloadFilter={"idNearRTRIC":"1"}&pageLimit=500&pointInTime=2021-04-21T00:00:00-0:00&indexpageNumber=2", "previousRecordsLink": "cps-temporal/api/v1/dataspaces/{dataspace-name}/anchors/{anchor-name}/history?after=2021-03-21T00:00:00-0:00&maxSize&simplePayloadFilter={"idNearRTRIC":"1"}&pageLimit=500&pointInTime=2021-04-21T00:00:00-0:00&indexpageNumber=0", "records": [ { "timestamp": "12345677888892021-03-21T00:00:00-0:00", "dataspace": "my-dataspace", "schemaSet": "my-schema-set", "anchor": "my-anchor", "data": { "status" : "UP" } } ] } |
Pros:
- Provides the ability to cache data based on URL. It is not relevant as data can change based on when the API is called.
- Can provide links for the next record and previous record link in the response itself.
- pointInTime can have the current timestamp as the default value because we can add it in the next record and previous record link as a part of the response.
Cons
- If the payload filter condition becomes longer, it will not fit within the GET URL length limit.
Questions
- Should we disable caching or ask the user to pass the pointInTime parameter value?
Approach 2 - POST
As we are query data, POST is not intuitive but the search can be considered as a resource with different filtering criteria. We need only one API as the filtering criteria will be provided in the body, which allows the payload to be big if required.
API URL: cps-temporal/api/v1/filters?maxSize=1000&index=1
Request Body
Code Block |
---|
{
"dataspaceName": "dataspace-001",
"anchorName": "anchor-001",
"schemaSetName": "schemaset-001",
"after" : "2021-03-21T00:00:00-0:00",
"pointInTime": "2021-04-21T00:00:00-0:00", // mandatory
"payload": {
"status": "UP"
}
} |
Response Body
Code Block |
---|
[
{
"timestamp": "1234567788889",
"dataspace": "my-dataspace",
"schemaSet": "my-schema-set",
"anchor": "my-anchor",
"data": {
"status": "UP"
}
}
] |
Pros:
- The payload can be big and we will still be able to support it.
Cons
- No right place to pass pagination related parameters ( maxSize & index)
- Less user-friendly than the GET approach.
- Using POST for query will break many rest principles.
Open Items
...