"Python is a programming language that lets you work more quickly and integrate your systems more effectively." - https://www.python.org/

Tips and techniques

  • Don't use assert statements for regular validation. assert statements can be disabled at the interpreter level, which would vastly change the flow of your code if they were used widespread.

Variable names

  • _varname - Semi-private. Basically a convention that developers use to indicate that the scope of a variable is local, but this locality is not enforced by the interpreter.
  • __varname - Private variable in name, but not in logic or security. The interpreter mangles the name of the var to make it globally unique, but it is still globally accessible.
  • var_ - Used to get around shadowing built-in variable names. EG: list_ won't conflict with list()
  • __magic_method__ - See https://www.diveinto.org/python3/special-method-names.html
  • _ - Temp var, pretty common entity in programming. (eg: bash and perl both support this too.)

Virtual Environments

Virtual environments isolate your project away from the system's python interpreter and modules, so you can have full control over what code is available to your project. This makes it easy to develop, debug, and deploy to a new system. It's basically always a good idea to use a virtual environment. You will thank yourself later by learning this one up front.

Virtual environments using venv

Creating a venv

echo "venv" >> .gitignore
virtualenv venv
. venv/bin/activate
pip install requests
pip freeze > requirements.txt
# write code, interact with it, whatever

Recreating a venv

virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
# write code, interact with it, whatever

Use venv to work around missing pip

This is mostly useful for installing for your user, since if you can't install pip you won't be able to install into system-wide locations.

virtualenv venv --system-site-packages && venv/bin/pip install --user "$PACKAGENAME" && rm -rf venv

Virtual environments with poetry

poetry is the new-school 2019 way of doing virtual environments. poetry stores its requirements in the new standard pyproject.toml file, and keeps the virtual environment stored outside of the current directory.

Creating a virtual environment using poetry

cd project_dir
poetry init
# walk through the dialogue
poetry add bpython boto3
poetry shell  # this spawns a subshell with the new python environment
# interact with your python environment

Import module from absolute path

import module # from somedir


Verbose environment var


# or...
python -v pip search beets

Follow the flow of a python script

This is equivalent to bash -x / bash -o xtrace, but is probably even more useful because it prefixes the name of the file and the line number to what is actually being executed, which aids in debugging large projects.

python -m trace --trace foo.py

You can get the equivalent output for a single function with:

import trace
tracer = trace.Trace(trace=True)
tracer.runfunc(some_func_name, 'some_func arg1', 'some_func arg2')

Enter an interactive prompt after script ends


This works when your code causes an exception, but none of your code will actually be executed, you will simply be dropped into a shell, which is not very useful.

# or...
sudo python -i ./ps_mem.py

Enter a python terminal arbitrarily


import pdb; pdb.set_trace()

In python 3.6+ you can simply insert breakpoint()

This drops you into a pdb shell. This is not the same as a full python REPL. To get a python REPL, type interact. After you have inspected the current state, you can type continue.

Alternatively there is the web-pdb package which allows you to debug via a web browser using web_pdb.set_trace

for var in dir():
    print "Debug: {0} = {1}".format(var,eval(var))

Inspect things

>>> import inspect
>>> inspect.getargspec(inspect.getargspec)
ArgSpec(args=['func'], varargs=None, keywords=None, defaults=None)

Create an http server using PWD as document root

python3 -m http.server 9980


python2 -m SimpleHTTPServer 9980

Discover the location for pip --user installs

echo $(python -m site --user-base)/bin

Add python's pip install --user bin path to PATH

# ~/.bash_profile
if PYTHON2_USER_BASE=$(python2 -m site --user-base 2>/dev/null) ; then PATH="${PYTHON2_USER_BASE}/bin:${PATH}" ; fi
if PYTHON3_USER_BASE=$(python3 -m site --user-base 2>/dev/null) ; then PATH="${PYTHON3_USER_BASE}/bin:${PATH}" ; fi

Show currently installed versions for all packages in requirements.txt

This can be used to update a requirements.txt file to the exact installed version

pip freeze | grep -f <(grep -o '^\w\+' requirements.txt)

Or auto pin to the current major version

pip freeze |
  grep -f <(grep -o '^\w\+' requirements.txt) |
  sed 's/==/~=/' |
  awk -F. '{print $1 "." $2}'

Common date operations

None of these examples deal with timezones.

Get the current time and date

>>> from datetime import datetime
>>> datetime.now().strftime("%s")
'1572039830'  # unix timestamp as a string
>>> datetime.now().strftime("%F")
'2019-10-25'  # abribrarily formatted timestamp string
>>> datetime.now()
datetime.datetime(2019, 10, 25, 14, 49, 49, 175165)  # as a datetime object

Convert from unix timestamp

>>> from datetime import datetime
>>> datetime.utcfromtimestamp(1234567890)
datetime.datetime(2009, 2, 13, 23, 31, 30)  # to a datetime object
>>> datetime.utcfromtimestamp(1234567890).strftime('%F %T')
'2009-02-13 23:31:30'  # to a string, via datetime object formatting

Convert from datetime string

>>> from datetime import datetime
>>> datetime.strptime('2019-05-01', "%Y-%m-%d")  # For some reason you can't use '%F' to describe inputs. Same with %s.
datetime.datetime(2019, 5, 1, 0, 0)  # to a datestamp object
>>> datetime.strptime('2019-05-01 10:01:59', "%Y-%m-%d %H:%M:%S").strftime('%A %B %d, %Y')
'Wednesday May 01, 2019'  # to a string, via datetime object

Generate a flame graph

Until https://github.com/baverman/flameprof/pull/5 is merged you may have to edit the installed flameprof shell script, or just use python2.

pip3 install flameprof
python -m cProfile -o myscript.prof myscript.py
flameprof myscript.prof.prof > output.svg




Various links