- CPS-251Getting issue details... STATUS
Overview
CPS temporal data query will allow the user to fetch data based on multiple filtering criteria. This data can be used to create graphs and help the analytical system to take decisions.
Filtering based on data
As it is time-series data, it can be filtered based on three main criteria:
Datetime
- Data in last X hours
- Data after a particular DateTime ( To be supported in version 1)
- Data between a date range
- Last X network data
- DataTypes in CPS Core
Dataspace & schema-set ( To be supported in version 1)
- Dataspace & anchor ( To be supported in version 1)
- Dataspace & multiple anchors - To improve performance, if there is a need to fetch data for multiple anchors.
- Payload ( To be supported in version 1)
Based on a field or subsection in the payload, the entire path from the root must be provided. These criteria need schema-set to be fixed to search in the same set of anchors. The details of the format is defined later.
We will implement only the basic APIs first and will add complex ones when required.
Other Parameters
There are other parameters, listed below, that can have an impact on the response.
No | Name | Type | Default | Purpose | Example |
---|---|---|---|---|---|
1 | index | int | 0 | The query output can have many rows so it is important to limit the fetched data. To limit the number of records and to provide pagination, these two parameters can be used. "index" parameter represents the page number, starts with 0, and maxSize is the page size. | |
2 | maxSize | int | 1000 | ||
3 | pointInTime | DateTime | CurrentDateTime | Pagination does not work well if new data gets added which fulfils the search criteria. The user must provide this value to avoid this issue. |
Assumptions
- Temporal database stores full information in the payload ( CPS-192: Design data store for Temporal Service).
- Data in the response body will always contain the entire payload data.
Payload Filter Format
The filter can be provided on any field at any level, but it is important to provide the entire path from the root. The below mention JSON is used as an example to explain the filtering format
The filtering scenarios and the format to be used for each is defined below:
No | Type of field | To be search | Format |
---|---|---|---|
1 | Top-level field | idNearRTRIC | |
2 | The field inside another object | nestedObject → id | |
3 | String collection field at the top level | labels | |
4 | Field inside a collection | GNBDUFunction → idGNBDUFunction |
Proposed APIs
Currently, we are not supporting all the filtering criteria defined in the overview. The below list defined the query parameters which can be used for all the endpoints.
Data by Anchor Name
URL: GET /dataspaces/{dataspace-name}/anchors/{anchor-name}
Example: GET /dataspaces/{dataspace-name}/anchors/{anchor-name}?after=2021-03-21T00:00:00-0:00&payload=/device/port/interface[status=up and type=type1]&maxSize=500
Supported Query Parameter apart from other parameter
- after - timestamp ( ISO Format) to consider date created after the specified Timestamp
- payload - to define data filtering criteria
Date by Dataspace and Schema-set
URL: GET /dataspaces/{dataspace-name}/schema-sets/{schema-set}
Supported Query Parameter apart from other parameter
- after - timestamp ( ISO Format) to consider date created after the specified Timestamp
- payload - to define data filtering criteria
Example: GET /dataspaces/{dataspace-name}/schema-sets/{schema-set}?after=2021-03-21T00:00:00-0:00
No | Purpose |
---|---|
1 | Return data entries for an anchor after the specified Datetime, which matches the payload format. |
2 | Return data entries based on provided schema-set after the specified Datetime, which matches the payload format. |
Response Body
name | type | |
---|---|---|
nextRecordsLink | string | added only if there are remaining records to be fetched for the query. |
previousRecordsLink | string | added only if it is not the first set of records. |
records | list | contains one record for each timestamp that meets filtering criteria. It contains header information along with data. |
{ "nextRecordsLink": "cps-temporal/api/v1/dataspaces/{dataspace-name}/anchors/{anchor-name}?after=2021-03-21T00:00:00-0:00&maxSize=500&pointInTime=2021-04-21T00:00:00-0:00&index=2", "previousRecordsLink": "cps-temporal/api/v1/dataspaces/{dataspace-name}/anchors/{anchor-name}?after=2021-03-21T00:00:00-0:00&maxSize=500&pointInTime=2021-04-21T00:00:00-0:00&index=0", "records": [ { "timestamp": "1234567788889", // TODO change the format "dataspace": "my-dataspace", "schemaSet": "my-schema-set", "anchor": "my-anchor", "data": { "status" : "UP" } } ] }
Pros:
- Provides the ability to cache data based on URL. It is not relevant as data can change based on when the API is called.
- Can provide links for the next record and previous record link in the response itself.
- pointInTime can have the current timestamp as the default value because we can add it in the next record and previous record link as a part of the response.
Cons
- If the payload filter condition becomes longer, it will not fit within the GET URL length limit. - TODO mention it somewhere ( Assumptions / Limitations / ? )
Questions
- Should we disable caching or ask the user to pass the pointInTime parameter value?