The CD stack sends the Jenkins console output directly to Elasticsearch using the logstash plugin. For this reason scripted fields are used in Kibana to parse and derive fields on load time instead of using logstash.
The instructions to configure these fields and import the dashboard are as follows:
- Update mappings to support using message.keyword field.
To do so on the Elasticsearch (ES) node run the following curl command included in the attached file:
jenkins_elasticsearch_mappings.txt Add the following parameter to the elasticsearch.yml config file:
script.painless.regex.enabled: true
Restart ES
- In Kibana create the index patter "logstash-jenkins". Set the timestamp field to "@buildTimestamp"
- For that index pattern navigate to the scripted fields tab. Create a scripted field named "healthcheck". Language is painless. Type is string. Script is as follows:
def msg = doc['message.keyword'].value;
if (doc['message.keyword'].value != null) {
if (doc['message.keyword'].value =~ /Health Check/) {
if (doc['message.keyword'].value =~ /PASS/) {
return "PASS";
}
else {
return "FAIL";
}
}
} - Create another scripted field named "component". Language is painless. Type is string. Script is as follows:
if (doc['message.keyword'].value != null) {
if (doc['message.keyword'].value =~ /Health Check/) {
def m = /^(.+?)(API)?\sHealth Check.*$/.matcher(doc['message.keyword'].value);
if ( m.matches() ) {
return m.group(1)
}
else {
return "unknown";
}
}
}
return null; Import the following visualizations to Kibana:
jenkins cd visualizations.json
Import the following dashboard to Kibana:
jenkins cd dashboard.json
Troubleshooting:
If you get the following error in Kibana:
"[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting","bytes_wanted":0,"bytes_limit":0}}}}]},"status":400}
Execute the following request on the Elasticsearch VM:
# thanks Shane
curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "script.max_compilations_per_minute" : 50 } }'
{"acknowledged":true,"persistent":{},"transient":{"script":{"max_compilations_per_minute":"50"}}}
3 Comments
Michael O'Brien
Michael O'Brien
The following occurs only on the dashboard periodically - refreshes bring up the visualizations - viewing each visualization separately is ok
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","script_stack":["... rn s.get() == v;}compare(() -> { def msg = doc['me ..."," ^---- HERE"],"script":"boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { def msg = doc['message.keyword'].value;\nif (doc['message.keyword'].value != null) {\nif (doc['message.keyword'].value =~ /Health Check/) {\n if (doc['message.keyword'].value =~ /PASS/) { \n return \"PASS\";\n }\n else {\n return \"FAIL\";\n }\n}\n} }, params.value);","lang":"painless"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-jenkins","node":"7rPCLxSWSbqMnGAt6WK1qg","reason":{"type":"query_shard_exception","reason":"failed to create query: {\n \"bool\" : {\n \"must\" : [\n {\n \"query_string\" : {\n \"query\" : \"message:\\\"Health Check\\\"\",\n \"fields\" : [ ],\n \"use_dis_max\" : true,\n \"tie_breaker\" : 0.0,\n \"default_operator\" : \"or\",\n \"auto_generate_phrase_queries\" : false,\n \"max_determinized_states\" : 10000,\n \"enable_position_increments\" : true,\n \"fuzziness\" : \"AUTO\",\n \"fuzzy_prefix_length\" : 0,\n \"fuzzy_max_expansions\" : 50,\n \"phrase_slop\" : 0,\n \"analyze_wildcard\" : true,\n \"escape\" : false,\n \"split_on_whitespace\" : true,\n \"boost\" : 1.0\n }\n },\n {\n \"script\" : {\n \"script\" : {\n \"source\" : \"boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { def msg = doc['message.keyword'].value;\\nif (doc['message.keyword'].value != null) {\\nif (doc['message.keyword'].value =~ /Health Check/) {\\n if (doc['message.keyword'].value =~ /PASS/) { \\n return \\\"PASS\\\";\\n }\\n else {\\n return \\\"FAIL\\\";\\n }\\n}\\n} }, params.value);\",\n \"lang\" : \"painless\",\n \"params\" : {\n \"value\" : \"FAIL\"\n }\n },\n \"boost\" : 1.0\n }\n },\n {\n \"match_all\" : {\n \"boost\" : 1.0\n }\n },\n {\n \"range\" : {\n \"@buildTimestamp\" : {\n \"from\" : null,\n \"to\" : null,\n \"include_lower\" : true,\n \"include_upper\" : true,\n \"boost\" : 1.0\n }\n }\n }\n ],\n \"disable_coord\" : false,\n \"adjust_pure_negative\" : true,\n \"boost\" : 1.0\n }\n}","index_uuid":"qYeOQRvLStKoku8bz0f7Cg","index":"logstash-jenkins","caused_by":{"type":"script_exception","reason":"compile error","script_stack":["... rn s.get() == v;}compare(() -> { def msg = doc['me ..."," ^---- HERE"],"script":"boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { def msg = doc['message.keyword'].value;\nif (doc['message.keyword'].value != null) {\nif (doc['message.keyword'].value =~ /Health Check/) {\n if (doc['message.keyword'].value =~ /PASS/) { \n return \"PASS\";\n }\n else {\n return \"FAIL\";\n }\n}\n} }, params.value);","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"Not all paths provide a return value for method [lambda$0]."}}}}]},"status":400} at http://kibana.onap.cloud:5601/bundles/kibana.bundle.js?v=15554:229:30422 at Function.Promise.try (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:91:15709) at http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:91:15065 at Array.map (<anonymous>) at Function.Promise.map (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:91:15020) at callResponseHandlers (http://kibana.onap.cloud:5601/bundles/kibana.bundle.js?v=15554:229:30038) at http://kibana.onap.cloud:5601/bundles/kibana.bundle.js?v=15554:229:18276 at processQueue (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:38:23621) at http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:38:23888 at Scope.$eval (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:39:4619)Michael O'Brien
I'll also fix the following when I try to sort via the pie chart
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[script] Too many dynamic script compilations within one minute, max: [50/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting","bytes_wanted":0,"bytes_limit":0}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"logstash-jenkins","node":"7rPCLxSWSbqMnGAt6WK1qg","reason":{"type":"query_shard_exception","reason":"failed to create query: {\n \"bool\" : {\n \"must\" : [\n {\n \"query_string\" : {\n \"query\" : \"message:(\\\"Health Check\\\" AND \\\"PASS\\\")\",\n \"fields\" : [ ],\n \"use_dis_max\" : true,\n \"tie_breaker\" : 0.0,\n \"default_operator\" : \"or\",\n \"auto_generate_phrase_queries\" : false,\n \"max_determinized_states\" : 10000,\n \"enable_position_increments\" : true,\n \"fuzziness\" : \"AUTO\",\n \"fuzzy_prefix_length\" : 0,\n \"fuzzy_max_expansions\" : 50,\n \"phrase_slop\" : 0,\n \"analyze_wildcard\" : true,\n \"escape\" : false,\n \"split_on_whitespace\" : true,\n \"boost\" : 1.0\n }\n },\n {\n \"script\" : {\n \"script\" : {\n \"source\" : \"boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { def msg = doc['message.keyword'].value;\\nif (doc['message.keyword'].value != null) {\\nif (doc['message.keyword'].value =~ /Health Check/) {\\n if (doc['message.keyword'].value =~ /PASS/) { \\n return \\\"PASS\\\";\\n }\\n else {\\n return \\\"FAIL\\\";\\n }\\n}\\n} }, params.value);\",\n \"lang\" : \"painless\",\n \"params\" : {\n \"value\" : \"FAIL\"\n }\n },\n \"boost\" : 1.0\n }\n },\n {\n \"script\" : {\n \"script\" : {\n \"source\" : \"boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { if (doc['message.keyword'].value != null) {\\nif (doc['message.keyword'].value =~ /Health Check/) {\\n def m = /^(.+?)(API)?\\\\sHealth Check.*$/.matcher(doc['message.keyword'].value);\\n if ( m.matches() ) {\\n return m.group(1)\\n }\\n else {\\n return \\\"unknown\\\";\\n }\\n}\\n}\\nreturn null; }, params.value);\",\n \"lang\" : \"painless\",\n \"params\" : {\n \"value\" : \"Basic A&AI\"\n }\n },\n \"boost\" : 1.0\n }\n },\n {\n \"match_all\" : {\n \"boost\" : 1.0\n }\n },\n {\n \"range\" : {\n \"@buildTimestamp\" : {\n \"from\" : null,\n \"to\" : null,\n \"include_lower\" : true,\n \"include_upper\" : true,\n \"boost\" : 1.0\n }\n }\n }\n ],\n \"disable_coord\" : false,\n \"adjust_pure_negative\" : true,\n \"boost\" : 1.0\n }\n}","index_uuid":"qYeOQRvLStKoku8bz0f7Cg","index":"logstash-jenkins","caused_by":{"type":"general_script_exception","reason":"Failed to compile inline script [boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { def msg = doc['message.keyword'].value;\nif (doc['message.keyword'].value != null) {\nif (doc['message.keyword'].value =~ /Health Check/) {\n if (doc['message.keyword'].value =~ /PASS/) { \n return \"PASS\";\n }\n else {\n return \"FAIL\";\n }\n}\n} }, params.value);] using lang [painless]","caused_by":{"type":"circuit_breaking_exception","reason":"[script] Too many dynamic script compilations within one minute, max: [50/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting","bytes_wanted":0,"bytes_limit":0}}}}]},"status":400} at http://kibana.onap.cloud:5601/bundles/kibana.bundle.js?v=15554:229:30422 at Function.Promise.try (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:91:15709) at http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:91:15065 at Array.map (<anonymous>) at Function.Promise.map (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:91:15020) at callResponseHandlers (http://kibana.onap.cloud:5601/bundles/kibana.bundle.js?v=15554:229:30038) at http://kibana.onap.cloud:5601/bundles/kibana.bundle.js?v=15554:229:18276 at processQueue (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:38:23621) at http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:38:23888 at Scope.$eval (http://kibana.onap.cloud:5601/bundles/commons.bundle.js?v=15554:39:4619)