Python 3

Installation

Probably the more straightforward way to download is by installing natively. Modern Linux systems today tend to come with a pre-packaged version of Python in order to run some userspace pre-packaged programs (ufw is one).

To avoid conflicts with system binaries, Python versions can be managed via pyenv instead. Some installation instructions for Ubuntu 20.04 and 22.04 LTS:

# 1. Pull required binaries into build environment, needed to build python
#    See: https://github.com/pyenv/pyenv/wiki
sudo apt-get update
sudo apt-get install make build-essential libssl-dev zlib1g-dev libbz2-dev \
libreadline-dev libsqlite3-dev wget curl llvm libncursesw5-dev xz-utils tk-dev \
libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

# 2. Ensure git's autocrlf is set to false
#    See: https://github.com/pyenv/pyenv/issues/1725#issuecomment-724689291
#    Symptoms include: `/usr/bin/env: ‘bash\r’: No such file or directory` when running pyenv
git config --global core.autocrlf false

# 3. Run the automated installer
#    See: https://github.com/pyenv/pyenv-installer
curl -L https://github.com/pyenv/pyenv-installer/raw/master/bin/pyenv-installer | bash

# 4. Add scripts to shell to enable pyenv shims
#    For Bash, run the following commands:
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(pyenv init -)"' >> ~/.bashrc
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.profile
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.profile
echo 'eval "$(pyenv init -)"' >> ~/.profile
echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
echo 'command -v pyenv >/dev/null || export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile

# 5. Restart shell
exec "$SHELL"

# 6. Install Python version, e.g. 3.10.4
pyenv install 3.10.4

# 7. Set Python version as global version to use
     To unset, set it to 'system' insetad.
pyenv global 3.10.4

One great benefit of using pyenv is the ability to seamlessly transition between virtual environments when moving into a project directory, simply by creating a virtual environment and assigning a Python version to it. Multiple versions can be assigned as well.

On Windows platforms, Python should be uninstalled manually via the "Add & Remove Programs" settings page, instead of the usual pyenv uninstall .... The former seems to handle additional processes like removing entries from the registry, while the latter only deletes the program directory.

For OpenSUSE

As of 2023-07-30:

sudo zypper install gcc automake bzip2 libbz2-devel xz xz-devel openssl-devel ncurses-devel readline-devel zlib-devel tk-devel libffi-devel sqlite3-devel gdbm-devel patch make findutils

If failing with OpenSSL library not installed issue, common build problem page on wiki suggests to pass the library and header directory paths directly. For OpenSUSE 14.1, this translates to:

CPPFLAGS="-I/usr/include/openssl" LDFLAGS="-L/usr/lib64" pyenv install 3.10.6

Scripts

Small scripts to park here, until I manage to get my boilerplate library properly setup:

#!/usr/bin/env python3
# Rename files in "Screenshot from 2022-12-07 09-23-06.png" format to "20221207_092306_" 
# Justin, 2022-12-12
 
"""
Old contents:
#!/bin/sh
ls | grep -E "Screenshot .* ([0-9]{4})" | sed -r "s/([0-9]+)-([0-9]+)-([0-9]+) ([0-9]+)-([0-9]+)-([0-9]+)/\1\2\3_\4\5\6/g"
# TODO: Not complete!
"""
 
import pathlib
 
for path in pathlib.Path().glob("Screenshot from *.png"):
 
    # Extract all numbers
    target = "".join([c for c in str(path) if "0" <= c <= "9"])
    target = target[:8] + "_" + target[8:] + "_.png"
 
    # Rename
    path.rename(path.with_name(target).absolute())

setup.py?

PEP517 in a nutshell:

PS C:\Users\Justin> pip install git+https://github.com/pyuxiang/boiler.git
Collecting git+https://github.com/pyuxiang/boiler.git
  Cloning https://github.com/pyuxiang/boiler.git to c:\users\justin\appdata\local\temp\pip-req-build-rzbbiuw0
  Running command git clone --filter=blob:none --quiet https://github.com/pyuxiang/boiler.git 'C:\Users\Justin\AppData\Local\Temp\pip-req-build-rzbbiuw0'
  Resolved https://github.com/pyuxiang/boiler.git to commit 63522bae91982a15919d07160f68d4068169c336
  Preparing metadata (setup.py) ... done
Requirement already satisfied: numpy in c:\users\justin\.pyenv\pyenv-win\versions\3.11.1\lib\site-packages (from boiler==0.0.5) (1.24.1)
Installing collected packages: boiler
  DEPRECATION: boiler is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for boiler ... done
Successfully installed boiler-0.0.5

Saving Python history file, without outputs, to copy into script:

import readline
readline.write_history_file("history.txt")

For interactive mode, try:

user:~$ python3 -i -c "from {{MODULE}} import *"

To expose all variables from single function (for quick scripting purposes), use globals().update(locals()).

A post-install script from ''freqcd.c''

_postinstall.sh

#!/bin/sh
# Stand-in script for the 'freqcd' binary. Do not modify.
#
# This stub is required for compliance with pyproject.toml specification,
# when the 'freqcd' binary has not yet been compiled. For convenience,
# this script will automatically trigger the compilation and then proceed
# to run the desired binary :)
#
# Works only on Linux, with 'ps', 'awk' and 'python3' dependencies.
 
# Gets the full call to current script, then retrieves program path
# e.g. /bin/sh /home/justin/.pyenv/versions/3.10.6/bin/freqcd -x ...
PROGRAM=$(ps -q $$ -o args= | awk -F ' ' '{print $2}')
 
# Get root directory of installed library, where postinstall script is located
ENTRYPOINT=$(python3 -c "import importlib; print(importlib.util.find_spec('fpfind').origin)")
INSTALL_ROOT=$(dirname "$(readlink -f ${ENTRYPOINT})")
 
# Keep a record of program path, in case one desires a manual recompilation;
# will be created in-situ only in an editable installation.
rm -f ${INSTALL_ROOT}/freqcd_path \
    && echo "${PROGRAM}" > ${INSTALL_ROOT}/freqcd_path
 
# Trigger post-install compilation and write directly to target
cd ${INSTALL_ROOT} && ./_postinstall ${PROGRAM} \
    || { echo "'freqcd.c' failed to compile... not running."; exit 1; }
 
# Pass arguments and directly execute
exec "${PROGRAM}" "$@"
 
 
 
# Explanation of this script as follows.
#
# We go through this trouble of adding console scripts via the build backend
# is so that the backend can manage its installation/uninstallation, as
# opposed to manual aliases/symlinks.
#
# Some issues need to be addressed before this will work:
#   1. The use of 'pyenv' for Python management causes the binaries
#      to be called as a shim, i.e. the call occurs within a nested script
#      that first sources the Python environment dynamically. The actual
#      binary location needs to be identified.
#   2. The binaries, during the library installation process, are copied
#      over from the repo directly, and is not a soft link. The binary
#      itself will thus need to be replaced.
#
# Using the 'importlib' mechanism, i.e.
#
#    import importlib; print(importlib.util.find_spec('fpfind').origin)
#
# does not work when the library is installed as an editable package,
# although it does work otherwise. The same applies to the pip discovery
# method, with the additional 'pip' dependency, i.e.
#
#    python3 -m pip show ${LIBNAME} | grep 'Location' | awk -F ': ' '{print $2}'
#
# Instead, we rely on the fact that this script will be triggered by 'sh',
# with the desired path supplied as an argument. Extracting the path is thus
# a simple matter of a 'ps' + 'awk' hack.
#
# A small side-note, use of '$PPID' to extract the parent path will not work,
# which may end up deferring to the shell emulator's process instead.
#
# Soft links could have been used instead of directly copying the binary,
# so that editable installations can still freely recompile, independently
# of the Python library install flow. However, Git will continue to track
# this file (potentially leading to accidental binary commits), and requiring
# users to run 'git-skip-worktree' manually is not ideal. A security concern
# may also arise when the editable installation is publicly modifiable.
#
# We instead the binary directly. To accomodate the need for updates, the
# path to the binary is provided during the compilation process, at:
#
#    'src/fpfind/freqcd_path'
#
# To use the soft link behaviour (at your risk), replace the code with:
#
#    cd ${INSTALL_ROOT} && ./_postinstall ${INSTALL_ROOT}/freqcd \
#        || { echo "'freqcd.c' failed to compile... not running."; exit 1; }
#    rm -f ${PROGRAM} && ln -s ${INSTALL_ROOT}/freqcd ${PROGRAM}

See libraries that are likely to be manually imported, i.e. libraries that are not dependencies:

user:~$ pip3 list --not-required

A list of popular libraries on my side:

requirements.txt

# General stuff
arrow
joblib
matplotlib
numpy
pandas
pyserial
requests
requests-cache
scipy
tqdm
uncertainties

# Linting
black
isort

# S15 stuff
setuptools
Cython
fpfind @ git+https://github.com/s-fifteen-instruments/fpfind.git
S15lib @ git+https://github.com/s-fifteen-instruments/pyS15.git

# Personal
kochen @ git+https://github.com/pyuxiang/kochen.git

Running a single file code remotely can be done by passing the script via stdin over SSH (assuming the dependency libraries are already available on the remote side), then reading the output back. If a more complex data structure needs to be passed, one could use a message passing service or construct... or just pickle it if it's back to another Python module. See:

# Remote script, "producer.py"
if __name__ == "__main__":
    import contextlib, pickle, sys
    with contextlib.redirect_stdout(None):
        result = myfunc()
    sys.stdout.buffer.write(pickle.dumps(result))
 
# Local script, "consumer.py"
import pickle, sys
result = pickle.loads(sys.stdin.buffer.read())

user:~$ cat producer.py | ssh <HOST> python3 - <ARGS> | ./consumer.py
 
# If using pyenv, then may need to use a login shell to trigger .bashrc source
user:~$ cat producer.py | ssh <HOST> 'bash -lc "python3 - <ARGS>"' | ./consumer.py

The Python interpreter performance can be improved by setting compile-time optimization flags, as shown in this page, e.g.

env PYTHON_CONFIGURE_OPTS='--enable-optimizations --with-lto' PYTHON_CFLAGS='-march=native -mtune=native' pyenv install 3.10.6

Command-line scripts

Some libraries are designed to be installed as a Python module, with command line tools exposed. If the main method was defined, but no command line tool is available, consider mapping your own:

/usr/bin/aubio

#!/usr/bin/env python3
from aubio import cmd
if __name__ == "__main__":
    cmd.main()

In the case of Aubio, it was just that the commands are exposed only after restarting the shell...

uv

uv is fast becoming one of my favorite package + Python managers, by virtue of how blazingly fast it is. It also caches existing compilations so there's no need for manual building which is what pyenv does. Not really a good idea to expect other users to follow though (need to learn the extra uv commands), so the way to transparently add Python:

user:~$ uv python install 3.12
user:~$ uv venv
 
# Hide virtual environment name by setting to empty string
user:~$ sed -i -E 's/(export VIRTUAL_ENV_PROMPT)/\1=""/g' .venv/bin/activate
 
# Source Python installation
user:~$ echo '. "$HOME/.venv/bin/activate"' >> .profile
user:~$ uv pip install pip

Restarting the shell will result in a base Python + Pip, just like pyenv, but with users none the wiser :)