CONTENT
| Table of Contents |
|---|
...
MOTIVATION
The scripts where created mainly to ease the work of the doc team. Two tasks are supported:
- Automate the work in case multiple content shall be "moved" from the DeveloperWiki to ReadTheDocs (c2m.sh)
- Analyse warning messages of the doc build process, identify related modules/projects/rst-files and ease the debug process (warnstats.sh)
They Both scripts can be found in the doc repo/tools directory contained in the doc /tools directoryrepo. Please check also Readme README.md and the header of both scripts for additional information.
...
Of course ... your feedback is highly welcome.CONTENT
| Table of Contents |
|---|
...
c2m.sh
DESCRIPTION
c2m automates additional tasks required in case you want to export and convert a set of wiki pages. The export and first conversion to markdown is done by confluence2md, provided by viaboxx.
c2m processes a list of (to be exported) wiki pages, creates corresponding export directories, exports and converts pages (in various formats if required), opens an editor and cleans up afterwards. c2m checks also for problematic content in the export and creates a warning in case of detection.
USAGE
c2m.sh <your-page-list>
Before executing copy the confluence2md-2.1-fat.jar file from Maven repository to your working directory otherwise it won't work
ISSUES
- markdown (md) output of confluence2md contains sometimes tags that are somehow "merged" with the topic headline; manual edit is required here
OPEN
- confluence2md does not support all of the currently used confluence page types (structured-macros) - result for unsupported pages is "not satisfying"; enhancements (java) are required
- workaround: copy the needed content to a new "blanc page" without using problematic macros and export it instead of the original page
- opt: toc creation in root document in case you export a tree of documents to separate files
REQUIRED
- pandoc
- retext
- confluence2md from https://repo1.maven.org/maven2/de/viaboxx/markdown/confluence2md/2.1/
- java (older version for confluence2md)
ADDITIONAL FILES
- example.pagelist: an example pagelist to demonstrate functionality
SEE ALSO
EXAMPLE PAGELIST
- example pagelist (field descriptions below); it uses the delimiter "|" for the four fields per line
- copy/paste page id and title from wiki; to get the wiki page_id you have to login to the wiki, open the page and choose e.g. the history
- depth: use depth to follow down the child-pages hierarchy if required: -1=infinte, 0=no children, #=number of child-pages to follow
- every hierarchy "0" entry will lead into the creation of a dedicated working directory where the page and child-pages are stored
- for better readability you can add spaces to the list, but use "|" as a delimiter
- lines starting with a # are filtered by c2m
# hierarchy | page_id | page_title | depth0 0 | 1018748 | ONAP Portal | 0
1.1 1 | 1018759 | ONAP Portal for users | 0
1.2 2 | 1018762 | ONAP Portal for administrators | 0
1.2.1 1 | 1018764 | Admins | 0
1.2.2 | 1018811 | Users | 0
1.2.3 | 1018821 | Portal Admins | 0
1.2.4 4 | 1018826 | Application Onboarding | 0
1.2.5 5 | 1018832 | Widget Onboarding | 0
1.2.6 | 1018835 | Edit Functional Menu | 0
1.2.7 | 16004953 | Portal Microservices Onboarding | 0
in case you want to export to only one single output page (that contains all child-pages of the above example) use:
0 0 | 1018748 | ONAP Portal | -1
...
warnstats.sh
DESCRIPTION
warnstat warnstats helps to find the onap modules (projects) and rst-files which are responsible for the most warnings during the documentation build process. It requires a tox build logfile, parses it line by line, prints out some statistics and provides links to the local rst file, its html version, the related link to readthedocs and as well the doc8 test result for the rst.
"rst html web doc8_(00000)" links can be opened easily by using mouse-over/context menu functionality of bash. files will be opened in your browser and rst editor!
Number of warnings found in the logfile are shown as "nnnnn", e.g. "00129"
Number of doc8 warnings/errors in a rst file are shown as "doc8_(nnnnn)", e.g. "doc8_(00002)".
USAGE
warnstatst.sh <your-tox-build-logfile>
NOTES
create the required tox logfile when you locally build the onap documentation:
tox | tee tox-build-logfile
ADDITIONAL FILES
- example-tox-build-log: an example tox logfile to demonstrate functionality (created links in the warnstats output may not work if you use this example)
EXAMPLE OUTPUT
...
...
| language | bash |
|---|---|
| theme | FadeToGrey |
(onapdocs) user@ubuntu:~/onapdocs/doc/tools$ ./warnstats.sh ../tox-build.log ...
warnstats version 1.6.0 (2020-04-03)
...
html build directory ..... /home/user/onapdocs/doc/docs/_build/html
...
web base url ............. https://docs.onap.org/en/latest
...
doc8 command ............. doc8 --verbose
...
doc8 results directory ... /home/user/onapdocs/doc/tools/doc8_results
...
tox logfile .............. ../tox-build.log
...
00656 LINES WITH WARNING IN FILE '../tox-build.log'
...
################################################################################
...
~~~ MESSAGES LONG ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
################################################################################
...
00656 WARNINGS IN TOTAL WITH 169 UNIQUE MESSAGES
...
00129 | document_isnt_included_in_any_toctree
...
00056 | Title_underline_too_short_
...
00053 | Could_not_lex_literal_block_as_json__Highlighting_skipped_
...
00035 | Unexpected_indentation_
...
00029 | Block_quote_ends_without_a_blank_line_unexpected_unindent_
...
- - snip - - - - - - - - - - - - - -
...
00001 | Citation_Log_Architecture_is_not_referenced_
...
00001 | Citation_Eventlet_is_not_referenced_
...
00001 | Citation_Django_Document_is_not_referenced_
...
00001 | Citation_django_deploy_is_not_referenced_
...
00001 | Anonymous_hyperlink_mismatch_7_references_but_0_targets_
...
################################################################################
...
~~~ MESSAGES SHORTENED (FOR SIMPLE GROUPING) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
################################################################################
...
00656 WARNINGS IN TOTAL WITH 46 UNIQUE MESSAGES
...
00129 | document_isnt_in
...
00094 | Could_not_lex_li
...
00072 | Duplicate_explic
...
00056 | Title_underline_
...
00039 | Unknown_target_n
...
- - snip - - - - - - - - - - - - - -
...
00001 | Citation_Log_Arc
...
00001 | Citation_Eventle
...
00001 | Citation_Django_
...
00001 | Citation_django_
...
00001 | Anonymous_hyperl
...
################################################################################
...
~~~ MODULES ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
################################################################################
...
00656 WARNINGS IN TOTAL IN 37 MODULES
...
00123 | dcaegen2.git
...
00078 | ccsdk
...
00043 | vnfrqts
...
00043 | multicloud
...
00039 | dmaap
...
- - snip - - - - - - - - - - - - - -
...
00001 | vid.git
...
00001 | usecase-ui.git
...
00001 | so
...
00001 | logging-analytics
...
00001 | docs_use-cases
...
################################################################################
...
~~~ MODULES WITH RSTFILES ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
################################################################################
...
00656 WARNINGS IN TOTAL IN APPROX. 241 RST FILES
...
aaf | index.rst | rst html web doc8_(00008) | 00006
...
aaf | README.rst | rst html web doc8_(00000) | 00001
...
aaf | service.rst | rst html web doc8_(00140) | 00002
...
aaf | usage.rst | rst html web doc8_(00007) | 00006
...
aai | AAIRESTAPI_AMSTERDAM.rst | rst html web doc8_(00008) | 00001
...
- - snip - - - - - - - - - - - - - -
...
vnfrqts | VES_Registration_3_2.rst | rst html web doc8_(00076) | 00001
...
vnfsdk | csar-validation.rst | rst html web doc8_(00010) | 00002
...
vnfsdk | index.rst | rst html web doc8_(00003) | 00002
...
vnfsdk | release-notes.rst | rst html web doc8_(00009) | 00003
...
vnfsdk | VNFSDK-Marketplace-userguide-vendors.rst | rst html web doc8_(00002) | 00001
...
################################################################################
...
~~~ RSTFILES ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...
################################################################################
...
00656 WARNINGS IN TOTAL IN APPROX. 241 RST FILES
...
00030 | dmaap | Installation.rst | rst html web doc8_(00023)
...
00019 | ccsdk | workflow.rst | rst html web doc8_(00002)
...
00017 | dcaegen2.git | userguide.rst | rst html web doc8_(00051)
...
00016 | dcaegen2.git | mod-onboardingapi.rst | rst html web doc8_(00006)
...
00015 | appc.git | APPC_LCM_API_Guide.rst | rst html web doc8_(01121)
...
- - snip - - - - - - - - - - - - - -
...
00001 | aai | bulkApi.rst | rst html web doc8_(00011)
...
00001 | aai | AAIRESTAPI_DUBLIN.rst | rst html web doc8_(00542)
...
00001 | aai | AAIRESTAPI_CASABLANCA.rst | rst html web doc8_(00535)
...
00001 | aai | AAIRESTAPI_AMSTERDAM.rst | rst html web doc8_(00008)
...
00001 | aaf | README.rst | rst html web doc8_(00000)