2024-11-17
From language-specific tools like pip and conda to OS-level tools like apt and pacman, we rely on package management tools to handle dependencies when building projects. This article aims to summarize tips for using these tools.
If you frequently use package management tools, you might find inspiration in this article. However, this article does not guarantee comprehensiveness (it may not be the best perspective to understand them or the best practice to use them) and accuracy (please refer to the tool's documentation instead).
Some small tips:
pacman -S _package_name1_ _package_name2_ installs the package and its dependencies.
In the pacman database, all packages are categorized based on the reason for installation: explicitly installed (explicit) and installed as dependencies (deps). Use pacman -S --asdeps _package_name_ to manually install a package as a dependency.
You can query the database using pacman -Q options, such as -Qe to list all explicitly installed packages; -Qi
Consider the following example:
$ sudo pacman -S pipewire
error: failed to prepare transaction (could not satisfy dependencies)
installing libcamera (0.3.0-1) breaks dependency 'libcamera.so=0.2-64' required by libcamera-ipa
$ sudo pacman -S libcamera libcamera-ipa
installing libcamera (0.3.0-1) breaks dependency 'libcamera.so=0.2-64' required by pipewire
This indicates that the current version of libcamera (likely installed as a dependency) is simultaneously depended on by pipewire and libcamera-ipa. The new version of pipewire depends on the new version of libcamera, and upgrading pipewire must automatically upgrade libcamera, which would break the old version of libcamera-ipa (and this old version might be explicitly installed, so the package manager hesitates to upgrade it automatically). The reverse is also true, forming a version deadlock.
libcamera <- libcamera-ipa
\_ pipewire
-- means version tied
In such a scenario, you can only upgrade libcamera-ipa and pipewire simultaneously. In this case, the package manager knows that both packages need to be upgraded, thus providing a plan to upgrade all three packages simultaneously:
sudo pacman -S libcamera-ipa pipewire
[Update on December 30, 2024]
For version inconsistency issues in Arch Linux systems, it might be the result of a previous partial upgrade (Arch Linux does not support partial upgrades). The recommended solution is to use the -u (i.e., --sysupgrade) option of pacman -S to update the entire system:
sudo pacman -Syu pipewire libcamera libcamera-ipa
When installing packages on Ubuntu, you often encounter the error "Error 404... E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?" After running apt-get update and reinstalling, it succeeds. Why?
The description of apt-get update is "Used to re-synchronize the package index files from their sources. The indexes of available packages are fetched from the location(s) specified in _/etc/apt/_sources.list." In simple terms, the error is likely because the links stored in the local database have expired or do not have the latest version's address. apt-get update resolves this issue by fetching the latest package index from the remote.
This corresponds to the -y option of pacman, which "updates the local package database that pacman maintains and prevents pacman from trying to install & requesting an outdated version from mirror sites," and is the solution to "Error retrieving file from… the requested URL returned error 404."
[Updated on December 30, 2024]
For pacman on Arch Linux, when refreshing the local package database, you must update the entire system. (When refreshing the package database, always do a full upgrade with pacman -Syu.)
In other words, do not use commands like pacman -Sy package.
Some errors may be caused by outdated package versions that the package manager itself depends on. In such cases, try upgrading this dependency first. For example, if pacman reports "corrupted package - PGP signature," it might be due to an outdated version of archlinux-keyring. The solution is pacman -Sy --needed archlinux-keyring. (Updated on December 30, 2024: The reason this command does not include -u is that updating all packages in the system can cause some packages to report "corrupted package" errors during installation. Therefore, a partial upgrade to update archlinux-keyring must be done first, followed by a full system update.)
Take pacman -Qi dbus as an example, its output is:
Name : dbus
Version : 1.14.10-1
Description : Freedesktop.org message bus system
Architecture : x86_64
URL : https://wiki.freedesktop.org/www/Software/dbus/
Licenses : GPL custom
Groups : None
Provides : libdbus libdbus-1.so=3-64
Depends On : audit expat systemd-libs libaudit.so=1-64 libsystemd.so=0-64
Optional Deps : None
Required By : avahi bluez jack2 libdecor libnvme libpcap libpulse libteam pipewire pipewire-audio pipewire-pulse rtkit systemd wpa_supplicant xorg-server
Optional For : None
Conflicts With : libdbus
Replaces : libdbus
Installed Size : 911.37 KiB
Packager : Jan Alexander Steffens (heftig) <heftig@archlinux.org>
...
Install Reason : Installed as a dependency for another package
Dbus is an inter-process communication system, usually installed as a dependency of systemd (system and service manager). You can see that package dependencies can be packages or may also include the corresponding .so files (shared objects, shared libraries, dynamic link libraries). At the same time, what the package provides can also be packages and .so files. You can see which packages depend on this package and whether this package was explicitly installed via the command line or as a dependency.
Consider the following example:
>>> import openai
...
File "C:\miniconda3\envs\genir\lib\site-packages\pydantic_core\__init__.py", line 6, in <module>
from ._pydantic_core import (
ModuleNotFoundError: No module named 'pydantic_core._pydantic_core
In this environment, both pydantic and pydantic core are installed as dependencies. They have issues and need to be reinstalled or upgraded, remembering to include the packages that depend on them and were explicitly installed:
pip install pydantic pydantic_core openai
A Python distribution package is downloadable software. For example, pip install pkg downloads a distribution package named pkg.
A Python import package is a Python programming language module. The Python Modules documentation explains modules as "definitions placed in a file for use in scripts or interactive interpreters for maintainability of long programs," and the Python Packaging Glossary defines it as the "basic unit of code reusability," while the JavaScript documentation defines modules as "splitting JS programs into separate files in complex projects for import when needed." In a program, import pkg imports a module.
Note: In Python, an extension module is defined as a module written in lower-level languages like C, C++, etc.
Downloading a distribution package allows the use of the corresponding import package, but they may not have the same name. For example, the Pillow distribution package for image processing provides the PIL module. Assuming import foo means pip install foo is prone to errors and dangerous. Different distribution packages may provide import packages with the same name (e.g., forks from code repositories), and rarely, a distribution package may provide multiple import packages.
Packages on PyPI (Python Package Index) exist in two formats: Source Distribution (sdist) and Binary Distribution (also known as Wheel). The former looks like pip-23.3.1.tar.gz, and the latter looks like pip-23.3.1-py3-none-any.whl.
The .tar.gz archive in the source distribution package contains the source code and a PKG-INFO file that stores metadata. As a small experiment, you can download pip-24.3.1.tar.gz from Pypi, then extract it using python -m tarfile -e pip-24.3.1.tar.gz, and inspect the files within.
A binary distribution package contains the files needed for installation. Typically, for packages with extension modules, a Wheel contains platform-specific binaries, such as DLLs or .so files. For modules written purely in Python, a Wheel strips out files commonly found in sdist, such as tests and documentation. A Wheel is a zip file; as a small experiment, you can download pip-24.3.1-py3-none-any.whl from Pypi, then extract it using python -m zipfile -e pip-24.3.1-py3-none-any.whl pip-wheel. In the Wheel name, py3 represents the Python implementation, none indicates no dependency on a specific Python version, and any indicates no platform dependency.
Egg is a format deprecated in August 2023 (replaced by wheel).
pip can directly install distribution packages in sdist and wheel formats: python -m pip install {sampleproject-1.0.tar.gz, sampleproject-1.0-py3-none-any.whl}
python -m pip list shows which distribution packages are installed in the environment.
python -m pip show package-name displays information about a distribution package.
Refer to this official tutorial
Build frontends, such as pip and build, delegate the task of building distribution packages to build backends, such as setuptools.
Create the following file tree, and fill in appropriate content in example.py and pyproject.toml. Below, it is assumed that __init__.py is left empty.
packaging_tutorial/
├── LICENSE
├── pyproject.toml
├── README.md
├── src/
│ └── mypkg/
│ ├── __init__.py
│ └── example.py
└── tests/
Run python -m build in this folder to get a dist folder, which generates the distribution package.
Install the distribution package: python -m pip install ./dist/mypkg-0.0.1-py3-none-any.whl, and verify it: from mypkg import example; example.add_one(2);
To add a package with extension modules to Python, you can refer to this tutorial (its use of commands like python setup.py build may be outdated as of 2024).