Skip to content

python

"Python is a programming language that lets you work more quickly and integrate your systems more effectively." - https://www.python.org/

Tips and techniques

  • Don't use assert statements for regular validation. assert statements can be disabled at the interpreter level, which would vastly change the flow of your code if they were used widespread.

Variable names

  • _varname: Semi-private. Basically a convention that developers use to indicate that the scope of a variable is local, but this locality is not enforced by the interpreter.
  • __varname: Private variable in name, but not in logic or security. The interpreter mangles the name of the var to make it globally unique, but it is still globally accessible.
  • var_: Used to get around shadowing built-in variable names. EG: list_ won't conflict with list()
  • __magic_method__: See https://diveintopython3.net/special-method-names.html
  • _: Temp var, pretty common entity in programming. (eg: bash and perl both support this too.)

Virtual Environments

Virtual environments isolate your project away from the system's python interpreter and modules, so you can have full control over what code is available to your project. This makes it easy to develop, debug, and deploy to a new system. It's basically always a good idea to use a virtual environment. You will thank yourself later by learning this one up front.

Virtual environments using venv

Creating a venv
echo "venv" >> .gitignore  # optional
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
## write code, interact with it, whatever
deactivate
Use venv to work around missing pip

This is mostly useful for installing for your user, since if you can't install pip you won't be able to install into system-wide locations.

virtualenv venv --system-site-packages && venv/bin/pip install --user "$PACKAGENAME" && rm -rf venv

Virtual environments with poetry

poetry is the new-school 2019 way of doing virtual environments. poetry stores its requirements in the new standard pyproject.toml file, and keeps the virtual environment stored outside of the current directory.

Creating a virtual environment using poetry
cd project_dir
poetry init
## walk through the dialogue
poetry add bpython boto3
poetry shell  # this spawns a subshell with the new python environment
## interact with your python environment
exit

Import module from absolute path

sys.path.append('/Users/username/code/somedir')
import module # from somedir

Convert between character number and string

You can use these functions to convert ascii and unicode characters into their numeric representations and back. Technically, ord converts a unicode character into a unicode code point, and chr does the reverse.

>>> ord('🐍')
128013
>>> chr(128013)
'🐍'

Benchmarking

Show stats about call count and times

This example shows how to profile a pytest run, and then generate a stats.txt file showing stats sorted by total time:

python -m cProfile -o output.prof pytest -sv tests
cat <<EOF | python -m pstats output.prof > stats.txt
sort time
stats
EOF

Yes, that syntax is ugly, and yes, the stats module could use a better CLI, but it works. Creating a function to make the CLI interface better is left as an exercise for the reader.

Generate a flame graph

Until https://github.com/baverman/flameprof/pull/5 is merged you may have to edit the installed flameprof shell script.

pip3 install flameprof
python -m cProfile -o output.prof myscript.py
flameprof output.prof > output.svg

Debugging

Verbose environment var

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONVERBOSE

export PYTHONVERBOSE=1
## or...
python -v pip search beets

Follow the flow of a python script

This is equivalent to bash -x / bash -o xtrace, but is probably even more useful because it prefixes the name of the file and the line number to what is actually being executed, which aids in debugging large projects.

python -m trace --trace foo.py

You can get the equivalent output for a single function with:

import trace
tracer = trace.Trace(trace=True)
tracer.runfunc(some_func_name, 'some_func arg1', 'some_func arg2')

Enter an interactive prompt after script ends

https://docs.python.org/3/using/cmdline.html#envvar-PYTHONINSPECT

This works when your code causes an exception, but none of your code will actually be executed, you will simply be dropped into a shell, which is not very useful.

export PYTHONINSPECT=1
## or...
sudo python -i ./ps_mem.py

Enter a python terminal arbitrarily

https://docs.python.org/3/library/pdb.html

import pdb; pdb.set_trace()

In python 3.6+ you can simply insert breakpoint()

This drops you into a pdb shell. This is not the same as a full python REPL. To get a python REPL, type interact. After you have inspected the current state, you can type continue.

Alternatively there is the web-pdb package which allows you to debug via a web browser using web_pdb.set_trace

for var in dir():
    print("Debug: {0} = {1}".format(var,eval(var)))

Inspect things

>>> import inspect
>>> inspect.getargspec(inspect.getargspec)
ArgSpec(args=['func'], varargs=None, keywords=None, defaults=None)

Create an http server using PWD as document root

python3 -m http.server 9980

Discover the location for pip --user installs

echo $(python -m site --user-base)/bin

Add python's pip install --user bin path to PATH

## ~/.bash_profile
if PYTHON3_USER_BASE=$(python3 -m site --user-base 2>/dev/null) ; then PATH="${PYTHON3_USER_BASE}/bin:${PATH}" ; fi

Manage a requirements.txt file like a pro

Managing requirements.txt manually can lead to a variety of problems related to dependencies and package compatibility. The best way to manage the requirements.txt file is by using the pip-tools command pip-compile, which builds a requirements.in file into a requirements.txt.

In your requirements.txt you define only your direct dependencies:

pendulum
typer

Then you run pip-compile --upgrade requirements.in, which would create a requirements.txt file like:

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile requirements.in
#
click==8.1.3
    # via typer
pendulum==2.1.2
    # via -r requirements.in
python-dateutil==2.8.2
    # via pendulum
pytzdata==2020.1
    # via pendulum
six==1.16.0
    # via python-dateutil
typer==0.7.0
    # via -r requirements.in

You can also specify --generate-hashes to get a more reliable lockfile style result.

Show currently installed versions for all packages in requirements.txt

This can be used to update a requirements.txt file to the exact installed version.

pip freeze | grep -f <(grep -o '^\w\+' requirements.txt)

Or auto pin to the current major version

pip freeze |
  grep -f <(grep -o '^\w\+' requirements.txt) |
  sed 's/==/~=/' |
  awk -F. '{print $1 "." $2}'

Generate a TZ aware datetime using only the standard library

from datetime import datetime, timezone
TZ = datetime.now(timezone.utc).astimezone().tzinfo
datetime.now(TZ).isoformat(timespec='seconds')  # 2023-06-24T12:50:01-04:00

Common date operations

None of these examples deal with timezones.

Get the current time and date

>>> from datetime import datetime
>>> datetime.now().strftime("%s")
'1572039830'  # unix timestamp as a string
>>> datetime.now().strftime("%F")
'2019-10-25'  # abribrarily formatted timestamp string
>>> datetime.now()
datetime.datetime(2019, 10, 25, 14, 49, 49, 175165)  # as a datetime object

Convert from unix timestamp

>>> from datetime import datetime
>>> datetime.utcfromtimestamp(1234567890)
datetime.datetime(2009, 2, 13, 23, 31, 30)  # to a datetime object
>>> datetime.utcfromtimestamp(1234567890).strftime('%F %T')
'2009-02-13 23:31:30'  # to a string, via datetime object formatting

Convert from datetime string

>>> from datetime import datetime
>>> datetime.strptime('2019-05-01', "%Y-%m-%d")  # For some reason you can't use '%F' to describe inputs. Same with %s.
datetime.datetime(2019, 5, 1, 0, 0)  # to a datestamp object
>>> datetime.strptime('2019-05-01 10:01:59', "%Y-%m-%d %H:%M:%S").strftime('%A %B %d, %Y')
'Wednesday May 01, 2019'  # to a string, via datetime object

Install a package from git

This is great for locally developed packages. This also works with pipx, which will install tools into their own virtualenv.

pip install git+ssh://gitserver/path/repo.git@git-ref

Decorators

Modules