SONARQUBE SUPPORT WILL END FEB-20
Please make sure there are no components left in SonarQube
Please implement SonarCloud coverage drop fix explained below
Timeline of events
To prevent dependency issues, this migration is planned to happen in 2 stages.
Stage 1 will use hardcoded sonar goals to the needed plugin version to avoid waiting on dependencies from where this can be inherited.
Stage 2 will be a cleanup stage and work towards fixing the plugin in the dependencies like oparent and slowly migrate teams to it while removing hardcoded sonar goals.
STAGE 1
- RELENG → Will add the needed changes in ci-management for each project and run a test in the sandbox
- RELENG → will post the restults of the sandbox test in the 3rd column below.
- This is not reflecting if the Sonar report actually pass or failed a quality gate. Instead, this result is the outcome of running the Sandbox job using the changes in ci-management.
- TECHTEAM → Review the results and comments posted by Releng in column 4. PTLs will be mentioned explicitly to notify them that the next action is on their team.
- TECHTEAM&RELENG → Will work on any additional comments/questions/issues from column 4 and 5.
SONAR COVERAGE DROP TO 0%
Sonar Cloud made an update on the jacoco plugin where they dropped support for .exec file format. We need to use .xml instead.
Please look for more information in https://community.sonarsource.com/t/sonar-coverage-dropping-to-0-for-some-projects/18521/2
Some teams like APPC have upgraded their reports and coverage was restored for them. Please follow their examples: https://gerrit.onap.org/r/#/c/appc/parent/+/100156/, https://gerrit.onap.org/r/#/c/appc/+/100155/
Also, OPARENT has been updated with the fix too https://gerrit.onap.org/r/#/c/oparent/+/100168/ included in 3.0.0-SNAPSHOT
| Task | Add SonarCloud variables in project's yaml files | Sandbox test of my change (DO NOT CONFUSE WITH SONAR REPORT RESULT) | Review results | Comment on progress |
|---|---|---|---|---|
| Responsible | RelEng | Releng | PTL | |
| Estimated Duration | 5 min | 15 min to 1 hr depending | 10 min per report | |
| AAF | https://gerrit.onap.org/r/#/c/ci-management/+/98542/ | PASS | Jonathan Gathman Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Need to fix 0% coverage: |
| AAI | https://gerrit.onap.org/r/c/ci-management/+/98544 | PASS | James Forsyth Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Completed
Update 19 Feb 2020: Issues have been fixed. AAI repos all passing in sonarcloud |
| APPC | https://gerrit.onap.org/r/c/ci-management/+/98545 | PASS | Takamune Cho This is marked as "PASS" becuase the build actually produced the same result as what we had in SonarQube, but I know we still are facing coverage issues. Shall we merge this change to move to SonarCloud and continue our investigation there? Jessica Wagantall I +1 already. we still have to work with the issue. I have posted my last comment on IT-17899. Please take a look. thx |
Issue has been fixed. back to 80% |
| CCSDK | https://gerrit.onap.org/r/c/ci-management/+/98546 | PASS | Dan Timoney Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Completed Dan Timoney CCSDK repos now all show coverage >= 55%. |
| CLAMP | https://gerrit.onap.org/r/#/c/ci-management/+/98543/ | PASS | all looks ok. | Tested and working with JDK11 |
| CLI | https://gerrit.onap.org/r/c/ci-management/+/98547 | PASS | user-67d6f Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | completed with coverage 61 % (25 Feb 2020) https://sonarcloud.io/organizations/onap/projects/?search=cli |
| DCAE | https://gerrit.onap.org/r/c/ci-management/+/98548 | PASS | 2/13 - All component migration completed and 0% coverage issue resolved | |
| Demo | https://gerrit.onap.org/r/#/c/ci-management/+/98549/ | PASS | Morgan Richomme Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | |
| DMAAP | https://gerrit.onap.org/r/#/c/ci-management/+/98550/ | PASS | Mandar Sawant Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok |
3/18 - Done |
| ExtAPI | https://gerrit.onap.org/r/#/c/ci-management/+/98551/ | fail | Adrian OSullivan , the Sonar jobs have been failing since before the migration. Can we please address them? --> | Completed |
| Holmes | https://gerrit.onap.org/r/#/c/ci-management/+/98552/ | PASS | Guangrong Fu Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Need to fix 0% coverage: https://sonarcloud.io/dashboard?id=onap_holmes-common https://sonarcloud.io/dashboard?id=onap_holmes-engine-management https://sonarcloud.io/dashboard?id=onap_holmes-rule-management |
| Integration | No Sonar jobs to migrate | |||
| Logging | https://gerrit.onap.org/r/#/c/ci-management/+/98554/ | PASS | Prudence Au Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Need to fix 0% coverage: |
| Modeling | https://gerrit.onap.org/r/#/c/ci-management/+/98553/ | PASS | Hui Deng Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok Completed | Completed |
| MSB | https://gerrit.onap.org/r/#/c/ci-management/+/98555/ | PASS | Huabing Zhao Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok Completed | Completed |
| MultiCloud | https://gerrit.onap.org/r/#/c/ci-management/+/98556/ | PASS | Completed | Done |
| Music | https://gerrit.onap.org/r/#/c/ci-management/+/98557/ | Pass | Bharath Balasubramanian Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok Completed | |
| OOF | https://gerrit.onap.org/r/#/c/optf/cmso/+/104333/ | 1 fail, seems like a sandbox npm environment | Completed |
https://sonarcloud.io/dashboard?id=onap_optf-fgps |
| OOM | No Sonar jobs to migrate | |||
| O-parent | No Sonar jobs to migrate | |||
| Policy | https://gerrit.onap.org/r/#/c/ci-management/+/98559/ | PASS | DONE | COMPLETED |
| Portal | https://gerrit.onap.org/r/#/c/ci-management/+/98560/ | PASS | Manoop Talasila Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | COMPLETED |
| Portal SDK | https://gerrit.onap.org/r/#/c/ci-management/+/98568/ | PASS | Manoop Talasila Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | 'npm install' failed which also fails |
| SDC | https://gerrit.onap.org/r/#/c/ci-management/+/98561/ | PASS | Ofir Sonsino Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Completed |
| SDNC | https://gerrit.onap.org/r/#/c/ci-management/+/98562/ | PASS | Dan Timoney Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Completed |
| SO | https://gerrit.onap.org/r/#/c/ci-management/+/98563/ | PASS | Seshu Kumar Mudiganti Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok https://sonarcloud.io/dashboard?id=onap_so | Completed |
| UUI | https://gerrit.onap.org/r/#/c/ci-management/+/98564/ | PASS | xu ran Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Completed |
| VFC | https://gerrit.onap.org/r/#/c/ci-management/+/98565/ | PASS | Yan Yang Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok Completed | Completed |
| VID | https://gerrit.onap.org/r/#/c/ci-management/+/98566/ | PASS | Tech team to compare reports: SonarQube: https://sonar.onap.org/dashboard?id=org.onap.vid%3Avid-parent SonarCloud: https://sonarcloud.io/dashboard?id=onap_vid | Completed |
VNFSDK | https://gerrit.onap.org/r/#/c/ci-management/+/98567/ | PASS | Weitao Gao Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | COMPLETED https://sonarcloud.io/organizations/onap/projects?search=vnfsdk |
| VVP | https://gerrit.onap.org/r/#/c/ci-management/+/98569/ | PASS | Gerrit ready, SonarCloud results pushed and waiting on committers to +1 my change if all looks ok | Completed
Fixed https://sonarcloud.io/dashboard?id=onap_vvp-validation-scripts For python projects, see https://gerrit.onap.org/r/c/vvp/validation-scripts/+/101356 for an example. |
STAGE 2
- Update oparent to upgrade the sonar plugin and Jacoco configuration to the latest plugin required by SonarCloud: https://sonarcloud.io/documentation/analysis/scan/sonarscanner-for-maven/
- Make an oparent release wit this change
- Migrate teams to the new oparent while also removing the hardcoded goals in their sonar ci-management changes from the above table
SonarCloud migration notes
Ready to migrate? Things to consider and challenges faced with the currently migrated teams
- Teams will still be using the same Jenkins jobs
- SonarCloud is based on SonarQube 8.0 with +- some feature and plugin differences
- SonarQube will stay around for some time, but eventually will be deprecated (mid next year)
- Warnings in Quality Gates are no longer applicable in SonarCloud
- Teams using the "org.sonarsource.scanner.maven" plugin, might face issues during the migration since this plugin calls SonarQube directly in older versions.
Testing the migration
The reports will now be posted under: https://sonarcloud.io/organizations/onap/projects
Jenkins jobs do not change.
14 Comments
Pamela Dragosh
Jessica Wagantall and Brian Freeman : integration oparent doesn't have sonar jobs, but it is a place where it defines JaCoCo settings. Do those need to be updated?
Jessica Wagantall
Thanks for noticing and for the reviews Pamela Dragosh, yes.. oparent is using sonar-maven-plugin which calls libraries from SonarQube and we need to remove it.
That is the reason why most of my tests are failing in the sandbox since it is inheriting.
I believe we need to do something like Taka did here https://gerrit.onap.org/r/#/c/appc/+/98197/4/pom.xml
Can anyone tell me if the same code applied in Oparent? I noticed Taka configured this plugin in two places too ..
Pamela Dragosh
Yes - we override the oparent definition in our repositories because we need to aggregate the reports. Why is Taka not aggregating his reports? That could be why his % is low. We aggregate and that gets our coverage up. In other words, if code is used by one sub-module for testing then it will get counted vs just relying on an individual JUnit to cover the whole class.
Jessica Wagantall
Thanks for pointing it out! Takamune Cho, Can we please define your reports for Sonar?
We still haven't been able to test APPC in SonarCloud since we are waiting for CCSDK-parent to remove their sonar-maven-plugin too, but maybe a similar reason is making APPC coverage drop in SonarQube?
Pamela Dragosh Brian Freeman If we fix oparent't pom.xml, it should not break dependnecies since they are pointing to a specific released version for Oparent right?
Will the right process be:
I understand this will cause some downtime for Sonar reports to be not produced meanwhile the migration is going, but at least we can still do it per component right?
By the sounds of it, it also seems that teams need to make more detailed configuration for their individual jacoco plugin.
Pamela Dragosh
Probably just need to update the SNAPSHOT for both elalto and master branch.
Teams can point to the SNAPSHOT.
Once we have multiple teams ported over and tested, then go ahead and release the oparent (at least from el alto branch, the master branch is in flux due to Java 11 upgrade).
Jessica Wagantall
Thanks Pamela Dragosh my biggest concern of using a Snapshot is what we normally face with the short life of Snapshots which might cause
some issues if the migration takes longer than few days.
I think we can make the release of Oparent from elalto and, as we planned before, migrate component by component. That way we save some steps
compare to migrating to the Snapshot and then re-migrate to the release version.
But if you want we can talk about it in the TSC and get more opinions.
Another concern is what you mentioned about aggregating the reports. I am not sure how to do that but I would definitely will need your help
explaining to the teams what to do if you don't mind.
Pamela Dragosh
I don't worry about SNAPSHOTs disappearing when we are full on in development. As a last resort, just do a re-merge.
All I'm saying is use the next few weeks to settle the oparent down and ensure it works with a good set of projects before bothering with a release. IMO its a moving target right now.
Takamune Cho
Please find comment on IT-17899 for more details. We got 60+ sonar java exceptions when sonarqube is trying to render the coverage report, over 2600 test cases are still successful during java verify build jenkins job. We do not use oparent, so I can't comment on oparent. Jessica Wagantall and I will try to migrate to sonar cloud to see if we can restore the test coverage back to 80%.
Jessica Wagantall
Takamune Cho thanks! In APPC's case, I had contacted Dan Timoney for help because we need CCSDK migrated first. I think they are depending on Oparent, is it?
Ram Krishna Verma
Login to new dashboard (https://sonarcloud.io/organizations/onap/projects) has no options for LF user/password. Do we need to create separate credentials for it?
Pamela Dragosh
Jessica Wagantall - this would be good to have. Previously I was able to login and create a list of favorites so I could easily keep track of the policy repositories.
Jessica Wagantall
I believe I forgot to mentioned here. The login needs to be done with your GithubID and matching pass
Seshu Kumar Mudiganti
Jessica Wagantall :
Could you please help me understand what needs to be done or the 0% coverage fix.
Henrik Andersson
I found this page with a very handy command to run Sonar from the command line: ExtAPI Developer guide#Sonar. Unfortunately it has not been updated with the new URL for SonarCloud instead of SonarQube. I have tried by just replacing the URL with the new one, https://sonarcloud.io/organizations/onap/projects, but I get an error:
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar (default-cli) on project a1-policy-management-service: Unable to execute SonarQube: Fail to parse entry in bootstrap index: <!DOCTYPE html><html lang="en"><head><meta charSet="utf-8"/><meta http-equiv="x-ua-compatible" content="ie=edge"/><meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no"/><style data-href="/styles.527f8da3001da95b4467.css">@import url(https://fonts.googleapis.com/css?family=Roboto:300,400,500,700,900);@import url(https://fonts.googleapis.com/css?family=Fira+Code:300,400,500);*,:after,:before{margin:0;paddin...
It would be relly nice if someone could also update this page to the new SonarCloud useage.