Background

Cloudify currently is built on top of Python 2.7. Formal support of Python2 has been scheduled (for the past 5 years) to go away on 1/1/2020.

Cloudify will be coming out with a version of the cloudify manager that runs on Python3 in 4Q2019. As part of this, they will need to convert all of their plugins to also run on Python3.

Unfortunately, any plugins that we have also need to be converted to Python3 in order to run on the new platform.

The Conversion Process

To prepare for this, we need to also convert our plugins to run under Python3. However, because of the timing we also need the code to continue running under Python2. To do this, we will follow these steps:

1. make sure tests run under Python2
2. convert code to be compatible with both Python2 and Python3
3. make sure tests continue to run under Python2 AND Python3

Make Sure that Python2 Tests Work

Before any conversion is done, make certain that you have tests for your code.

Many of the plugins are already set up with tox-based tests; these MUST be able to be run.

If tox-based tests do NOT exist, they should be created or an alternative form of testing platform needs to be provided.

The key point is: you need to have tests that can be run on your plugin code that is SEPARATE from it being run on a cloudify manager. That is, make certain that "tox test" runs cleanly for Python2.

Conversion to Both Python2 and Python3

There are a number of ways that this step can be accomplished. This is one way that seems to work fine:

Step 1: Create a Python3 Version of Your Files

Python3 comes with the 2to3 command. Use the 2to3 command with the -w option to create a Python3 version of each file:

For each file where differences were found that needed to be made, you will now find both a .bak file and an updated file. If there are no

Now rename each of your files that had differences to use ".py3" extensions:

Step 2: Merge the Two Versions

For each .py3 file, you will now create a file that has BOTH the Python2 AND the Python3 code present, with the differences clearly marked. The Linux GNU diff command has a nice facility to do this:

Your .py files will now have a series of blocks that look like this, using C preprocessor syntax:

These .py files are currently executable by NEITHER Python2 NOR Python3, so the next step is to deal with the differences to make them runnable by BOTH.

• There are some things allowed in Python2.7 that are mandated in Python3.
• There are other things that can be done in just the Python3 form if the right settings are provided.
• Or there are  things that can be done in just the Python2 form.
• There are things that must be done differently in the two languages, so a run-time choice is needed.

We will make use of each of these features.

Preparatory Statements

First step is to add these statements to the very top of each .py file:

If your .py file starts with a #! invocation statement, place those lines AFTER the shbang:

These lines do two things:

1. Allows the function version of the print statements to be used.
2. Gives us a value (USING_PYTHON2) that can be tested for at run time to determine if we are running Python2 or Python3.

Changing the #ifndef/#else/#endif Statements

You need to change blocks such as this:

to this. Make CERTAIN that the indentation is adjusted as well.

Yes, there is some cognitive dissonance for changing "#ifndef" to "if", but the rest of the generated code is set up properly to be able to quickly edit the code.

• Search for all instances of "#ifndef USING_PYTHON2" and change "#ifndef" to "if" and add a trailing ":".
• Search for all instances of "#else /* USING_PYTHON2 */" and change that to "else:".
• Remove all instances of "#endif /* USING_PYTHON2 */" from the code.
• Re-indent the lines in between.

• You should also search for "#ifdef USING_PYTHON2" and change that to "if not USING_PYTHON2:".


Import Statements

The way that imports of functions found in local files changes in Python3. So change blocks such as this:

to this. Make CERTAIN that the indentation is adjusted as well.

In some cases, standard library interfaces have changed. So change blocks such as this:

to this. Make CERTAIN that the indentation is adjusted as well.

Multiple such import statements can be combined together, as in:

Standard Library Differences

In some cases, library differences will be pointed out. Examples such as this:

to this:

Sometimes this can also be done using an inline if statement, as in:

It is your decision as to which is more readable.

Octal Constants

In Python2, octal constants can be prefixed with either "0" or "0o". In Python3, "0o" is mandated.

Code can use the new form exclusively. Change this

to just the new form:

has_key() vs 'in'

The has_key() method has been removed from Python3. Fortunately, the in keyword works in both languages. So change this

to only use the in keyword, as in this:

Print Statements vs Print Functions

The format of print statements have changed into functions in Python3. Because of the import statement we added in the Preparatory Steps above, we can use the print function format everywhere. Change this:

to only use the print() function, as in this:

Dictionary .keys() and .items() Return Iterators in Python3

In Python3, the dictionary .keys() and .items() methods return iterators instead of lists. There are two cases to be considered. In some cases a list must absolutely be used, such as passing to a format conversion. So the iterator can be converted to a list. This can be simplified even further by noting that converting a list to a list is a no-op, so the Python3 form of the code can be used in both languages. For example, change this:

to this:

When used in for loops, the two languages are compatible for the return of .keys() and .items(). So the Python2 form of the code can be used in both languages. For example, change this:

to just the Python2 form:

Additional Differences

All of the code samples above are meant to be examples of types of issues that have been discovered so far. Be sure to search the file for all instances of #ifndef and fix each occurrence.

If you discover additional types of changes that need to be discussed on this wiki pagr, please add them.

Clean Up Preparatory Statements

If there are no print() function calls in the code, this line can be removed from the Preparatory Statements at the top:

If there are no uses of USING_PYTHON2 in your code, this line can be removed from the top:

If there are no uses of "sys." in your code, you can also remove this line from the top:

Make Sure that Python2 AND Python3 Tests Work

To change your tox tests so that you are testing against both Python2 and Python3, a small change is needed to your tox.ini file. In particular, change this block from

to instead read

Now run "tox test" to execute your test modules once with Python2 and then again with Python3. (The final choice of py34 vs py36 vs py37 is yet to be determined.)

What to do When Python3 Tests Fail

If your Python3 tests fail within your code, your recourse is to fix your code. Just do it in a way that is compatible with both versions of Python.

Most of the cloudify mock libraries are compatible with Python3, but a few are not. If your Python3 tests fail because of an incompatibility in the cloudify mock libraries, until we get modified versions of the mock libraries we won't be able to run the tests with both versions of Python. You may be able to temporarily comment out some specific mock references and test the rest.