Python local-packages, à la npm node_modules

Kamal Mustafa - Nov 26 '19 - - Dev Community

Updates

Follow-up post where I made this into simple program called pipm.
**************** ## ***********

Firstly, this thing doesn't exists. Unlike npm or php composer, packaging tools in python by default will install to the global packages location first.

In npm or composer, if you don't specify npm install -g, then by default the package will be installed to folder node_modules in the project root. This quite intuitive, local first, then global.

In python's pip, if you don't want to install to the global packages directory, you can restrict it --user flag. For example:-

python3 -mpip install --user requests
Enter fullscreen mode Exit fullscreen mode

On linux, above command usually will install requests package to user's $HOME/.local/lib/python3.x/site-packages directory. You can see that this only "local" to the user, not project unlike npm node_modules above.

In python, the solution is to use virtualenv. To use virtualenv however, there are at least 2 schools of taught. The first, and also the majority is to put the virtualenv external to your project.

So you might create a directory in your $HOME/.virtualenvs to store all your virtual environment. Creating new virtualenv may look like this:-

python3 -mvenv $HOME/.virtualenvs/myproject
Enter fullscreen mode Exit fullscreen mode

Then you will 'activate' the environment in order to make it active:-

source $HOME/.virtualenvs/myproject/bin/activate
Enter fullscreen mode Exit fullscreen mode

From now on, when you run python it will actually invoke $HOME/.virtualenvs/myproject/bin/python. And any packages you install with pip, for example:-

python -mpip install requests
Enter fullscreen mode Exit fullscreen mode

will install the requests library to $HOME/.virtualenvs/myproject/lib/python3.x/site-packages/requests.

But doing all this manually found to be tedious, so there's tools like virtualenvwrapper, or modern tools like pipenv or poetry that automatically create virtualenv for you.

However I don't really like this approach. Forgetting to 'activate' the virtualenv or worst, activating wrong virtualenv for your project is one of the most painful debugging experience.

So I am in the second camp, who create the virtualenv inside our project dir. For example, if I have a project in $HOME/myproject, I'll do:-

cd $HOME/myproject
python -mvenv venv
Enter fullscreen mode Exit fullscreen mode

So above command will create the virtualenv in the project directory - $HOME/myproject/venv. From now on, whenever I need to invoke python, I'll run ./venv/bin/python from my project root. To install packages, I'll run:-

./venv/bin/python -mpip install requests
Enter fullscreen mode Exit fullscreen mode

That will install requests to ./venv/lib/python3.x/site-packages. Explicit is better than implicit.

If using poetry, you can achieve similar effect by having in your project's poetry.toml:-

[virtualenvs]
in-project = true
Enter fullscreen mode Exit fullscreen mode

Ok, back to the original topic. What if we don't want to use virtualenv? Fortunately, pip has a flag called --target where if specified it will install the package to the target's location. For example:-

python -mpip install -t local-packages requests
Enter fullscreen mode Exit fullscreen mode

(Using name local-packages as a play to site-packages)

Above command will instead install requests to the directory local-packages, which you can create in your project root, similar to node_modules.

Recent version of pip will also created the package's executable script (if they have one) in local-packages/bin directory. For example if we install package httpie, which provide executable script named http in local-packages/bin/http.

Unfortunately, if we try to run the executable script it will fail:-

./local-packages/bin/http 
Traceback (most recent call last):
  File "./local-packages/bin/http", line 6, in <module>
    from httpie.__main__ import main
ModuleNotFoundError: No module named 'httpie'
Enter fullscreen mode Exit fullscreen mode

This is not surprising as ./local-packages dir (which has our httpie packages) is not on python sys.path which is a list of paths where python look for modules to import. We can check it through the interactive console:-

python
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/kamal/.local/lib/python3.7/site-packages', '/usr/lib/python3.7/site-packages']
Enter fullscreen mode Exit fullscreen mode

To add additional path, we can specify PYTHONPATH environment variable:-

PYTHONPATH=$PWD/local-packages ./local-packages/bin/http https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Mon, 25 Nov 2019 22:36:19 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}
Enter fullscreen mode Exit fullscreen mode

So it works now. But there's one gotcha. If somehow we have httpie already installed in global or user site-packages, that one probably will get used instead of the one we have in our local-packages. This is bad and can cause another headache when debugging. And this is the reason we use virtualenv. But we don't want to use virtualenv, right?

Fortunately, python has another flag. The -S flag will disable processing site-packages. From the docs:-

Disable the import of the module site and the site-dependent manipulations of sys.path that it entails. Also disable these manipulations if site is explicitly imported later (call site.main() if you want them to be triggered).

But we run into another problem if we invoke python with the -S flag:-

PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 18, in <module>
    main()
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 10, in main
    from .core import main
  File "/home/kamal/python/testpip/local-packages/httpie/core.py", line 23, in <module>
    from httpie.client import get_response
  File "/home/kamal/python/testpip/local-packages/httpie/client.py", line 8, in <module>
    from httpie import sessions
  File "/home/kamal/python/testpip/local-packages/httpie/sessions.py", line 11, in <module>
    from httpie.plugins import plugin_manager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/__init__.py", line 10, in <module>
    from httpie.plugins.manager import PluginManager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/manager.py", line 2, in <module>
    from pkg_resources import iter_entry_points
ModuleNotFoundError: No module named 'pkg_resources'
Enter fullscreen mode Exit fullscreen mode

Let's try copying the pkg_resources module into our local-packages dir:-

cp -a /usr/lib/python3.7/site-packages/pkg_resources local-packages/
Enter fullscreen mode Exit fullscreen mode

No luck. We will find out that we need these modules to be available:-

  • pkg_resources
  • six
  • appdirs
  • packaging
  • pyparsing

And it turned out that all these modules actually part of setuptools, another beast in python packaging. So in short, in order to make our local-packages works, we need to install setuptools as well:-

python -mpip install -t local-packages setuptools
python -mpip install -t local-packages httpie
Enter fullscreen mode Exit fullscreen mode

And now invoking python with -S flag should work:-

PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Tue, 26 Nov 2019 00:15:54 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}
Enter fullscreen mode Exit fullscreen mode

And that's it. We finally manage to invoke python, free from global site-packages and using our local-packages dir to place all dependencies for our project.

End notes: In practice, we're using buildout to achieve the same objectives.

Image
14 Savvy Ways To Spend Extra Money All Terrain Self Propelled Wheelchair Budget
Image
The Self-Propelled Wheelchairs Mistake That Every Beginner Makes
Image
<h1>Gevelreiniging Zwolle</h1>
Image
This Is The History Of Personal Injury Lawsuit In 10 Milestones
Image
10 Things You Learned In Kindergarden Which Will Aid You In Obtaining Drip Coffee Brewer
Image
11 Ways To Completely Revamp Your Repairing Bifold Doors
Image
Graffiti verwijderen & aanbrengen antigraffiticoating
Image
The History Of Bifold Door Seal Replacement In 10 Milestones
Image
soledast2
Image
This Is How Best ADHD Medication For Adults Will Look Like In 10 Years Time
Image
The People Closest To What Is A Self Propelled Wheelchair Uncover Big Secrets
Image
Building a High-Quality Stock Report Generator with Node.js, Express, and OpenAI API

ai, openai, express, javascript
Image
14 Cartoons About Car Key Repair Cost That'll Brighten Your Day
Image
Some notes on SharedWorkers

javascript, sharedworker, vite
Image
Five Tools Everybody Is In The Electric Fireplace Built In Wall Industry Should Be Using
Image
Check Out: How How To Get ADHD Medication UK Is Taking Over And What To Do About It
Image
4 Reasons Why You Should move from Lombok to Java Records

codeproject, java, javadeserializations
Image
Here's An Interesting Fact Concerning Traffic Accident Lawyer Near Me
Image
Why Adding A Coffee Makers Drip To Your Life's Routine Will Make The Difference
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player