How to Do X In Python

I have been working with Python more and more professionally.

I am by no means a Python expert and this post is by no means exhaustive. This post is a living document that attempts to map other languages to the Python way of doing things and tries to quantify what the most "Pythonic" way of doing things is in different contexts. It will be updated as my understanding of the language improves.

Please feel free to let me know if there is a more Pythonic way of doing anything including examples that validate this claim.

Cheatsheet

Definition of Terms

• Convention: This feature/area is the conventional way of doing something in Python - it is not enforced by the language
• Language: This is a feature enforced by the language - an error will be generated if you violate this
• Pythonic: Indicates that the example used seems to conform to what people consider as Pythonic (based on examples and usages in large Python libraries)

Scope

This answer goes into these scoping conventions pretty well.

class Foo:

    def __init__(self, bar, name, age):
        # bar is a public class variable
        self.bar = bar

        # _name is protected meaning child classes can see and change this
        self._name = name

        # __age is private, only this class can see this
        # it is accessed from here using self.__age where it needs to be used
        self.__age = age

    # public method
    def fizz():
        print("Fizz")

    # protected method
    def _buzz():
        print("Buzz")

    # private method
    def __moo():
        print("Moo")

Back to TOC

Naming

• Method and variable naming convention

Files

Underscores are used to name multi word files:

fruit_repository.py

A single word file name would simply be the word and .py:

fruit.py

Back to TOC

Variables

Underscores are used to name multi word variables:

date_of_birth  = "12/12/2001"

A single word variable name would simply be the word:

name = "John"

Back to TOC

Enums

from enum import Enum

class Ordinal(Enum):
    NORTH="NORTH"
    SOUTH="SOUTH"
    EAST="EAST"
    WEST="WEST"

Getting the value behind the enum, in this case the string:

ordinal_str = some_ordinal_enum.value

Converting an enum value (in this case string) to the enum:

ordinal_from_value = Ordinal['NORTH']

Back to TOC

Methods

Underscores are used to name multi word method names:

def calculate_age():
    # do the calculations ...

A single word method name would simply be the word:

def bark():
    print("Woof")

Back to TOC

Testing

Paramterized Tests

The parameterized framework seems to have this covered.

For example to test permutations of the character N which can have zero or more whitespace characters before and after:

from parameterized import parameterized

class TestPersonType(TestCase):

    @parameterized.expand([
        ("lower_case_n", "n"),
        ("upper_case_N", "N"),
        ("spaced_N", "      N  "),
    ])
    def test_from_indicators__valid_N__outputs_correct_enum(self, name, valid_n_indicator):
        assert PersonType.from(valid_n_indicator) is PersonType.UNEMPLOYED

name is appended onto the end of the test function name after a sequence. For example my test panel looks as below after running this test:

test_from_indicators__valid_N__outputs_correct_enum_0_lower_case_n
test_from_indicators__valid_N__outputs_correct_enum_1_upper_case_N
test_from_indicators__valid_N__outputs_correct_enum_2_spaced_N

Back to TOC

Generative Tests

The hypothesis framework covers this.

For example if we are trying to test how a function handles inputs where lower case, upper case and any number of white spaces before/after the letter N is processed as can be seen here:

import re
from hypothesis import given, strategies

class TestPersonType(TestCase):

    @given(strategies.from_regex(re.compile("\\s*[Nn]\\s*"), True))
    def test_from_indicators__valid_N__outputs_correct_enum(self, valid_n_indicator):
        assert PersonType.from(valid_n_indicator) is PersonType.UNEMPLOYED

To see what this is doing simply change the above and add a print as below:

import re
from hypothesis import given, strategies

class TestPersonType(TestCase):

    @given(strategies.from_regex(re.compile("\\s*[Nn]\\s*"), True))
    def test_from_indicators__valid_N__outputs_correct_enum(self, valid_n_indicator):
        print(f"------------------ [{valid_n_indicator}]")
        assert PersonType.from(valid_n_indicator) is PersonType.UNEMPLOYED

This outputs something like the below:

------------------ [N]
------------------ […N]
------------------ […N]
------------------ [　…N  ]
------------------ […N]
------------------ [	N  ]
------------------ […N]
N]
------------------ [ N]
------------------ [ n]
------------------ [ n ]
------------------ [ n]
------------------ [n]
------------------ [  

  n      ]
------------------ [      n      ]
...

You can give these strategies as many parameters as you want for example if I want 2 inputs that are valid Ns:

VALID_N_STRATEGY = strategies.from_regex(re.compile("\\s*[Nn]\\s*"), True)
...
    @given(VALID_N_STRATEGY, VALID_N_STRATEGY)
    def test_from_indicators__valid_N__outputs_correct_enum(self, valid_n_indicator_1, valid_n_indicator_2):
...

Back to TOC

Miscellaneous

Dependency Management

pip, virtualenv and a requirements.txt file still seems to be the way people manage dependencies.

The general flow is:

• First Time:

  • Create the virtual environment in the root of the project: virtualenv venv
    • In Windows use the virtualenv-win wrapper.
  • Activate the virtual environment: source venv/bin/activate
  • Install requirements: pip install -r requirements.txt

• While Deving:

  • Activate the virtual environment: source venv/bin/activate
  • Do your dev
  • Do not forget to update your requirments: pip freeze > requirements.txt

Back to TOC

Install All Requirements Skipping Over Ones with Errors

If you keep getting an error when trying to install all requirements using pip install -r requirements.txt but you want to continue installing anyway run:

cat requirements.txt | xargs -n 1 pip install

You will still see errors but a dependency with errors will not hault the execution of the install for remaining dependencies.

Back to TOC

Reverse Engineer Dependencies

Use pipreqs. The basic usage of this is:

• pipreqs /path/to/project/root
  • This will look at the project's imports and output a requirements.txt file under /path/to/project/root/requirements.txt

It also has other options like looking for unused depdendencies and comparing the current requirements to what it thinks you do/don't need.

Back to TOC

Python Version Management

Use the excellent pyenv. A port of this (with less functionality) is available for windows called pyenv-win.

The general flow is:

• Find the version of Python you want: pyenv install -l
• Install it: pyenv install 3.7.8
• Tell the current project to use it: pyenv local 3.7.8
  • This creates a file in the project called: .python-version
• Set your global system python version to this: pyenv global 3.7.8
  • Be sure to add the following to your appropriate rc file:

eval "$(pyenv init -)"

Every time you switch to a new version of Python using pyenv you will need to pip install any global pips you use again.

Back to TOC

Main Method

Synchronous

if __name__ == '__main__':
    # your code to run in a main method here

Back to TOC

Asynchronous

import asyncio

# the name of the method can be anything but it has to be async
async def main():
    # await your async functions here

if __name__ == '__main__':
    asyncio.get_event_loop().run_until_complete(main())

Back to TOC

Daemonizing

I am not 100% sure if this is the Pythonic way of doing this but this approach has worked very well for me. It is a 2 part process:

• Create a makefile to run the command/s that start your app

For example:

run:
    cd your_package && pipenv install && pipenv run python -m your_package

• Create a service file that calls the correct makefile target:

[Unit]
Description=Your Awesome Python Service
After=network.target

[Service]
User=root
Restart=always
Type=simple
WorkingDirectory=/path/to/your/app
ExecStart=make run

[Install]
WantedBy=multi-user.target

Pretend this file was called my_service.service, copy this to /etc/systemd/system

Then to enable and start it run the following commands:

# enable the service (only need to do this once)
sudo systemctl enable my_service.service
# Start it up
sudo systemctl start my_service.service

# See the status of the service
sudo systemctl status my_service.service

# Tail the logs from the last 100 lines onwards
journalctl --unit=my_service.service -n 100 -f

The great thing with this approach is that your service will autorestart if it runs into unhandled or serious errors.

In one case I used this for an app that connected to various 3rd party endpoints that regularly failed, the app would simply restart and reconnect and it was good to go! I wanted this behaviour as there was no code changes I could do to cater for this. My app was also designed to pick up where it left off.

Back to TOC

Gotchas

Dashes in Filenames instead of underscores

You will not be able to import the module as you normally would - IDEs will not suggest these packages either.

Instead you will need to use __import__ as described in this answer:

python_code = __import__('python-code')

Back to TOC

Pass vs Continue

Pass is to prevent indentation errors. Take the following custom error class as an example:

class MyCustomError(Exception):

The above will give you an indentation expected error. To fix this we use pass as below:

class MyCustomError(Exception):
    pass

Back to TOC