Simple use case with local Python environment and Pipenv

Help
1

posting with approval on behalf of @mikhail.zholobov as this query could be useful to others as well :slight_smile:

I have some simple use case with local Python environment and I’d like to see if flox/nix can be useful here.

I have a couple of python scripts which I want to run locally in isolated environment with required Python packages (“xlsx2csv” in particular).
The scripts themselves are not a package, so I guess there is nothing to “build” there.

==> 1. Project overview

$ ls -l
total 32
-rw-r--r--@ 1 legal staff 168 Jan 6 2022 Pipfile
-rw-r--r--@ 1 legal staff 1771 Jan 6 2022 
Pipfile.lock
-rwxr-xr-x@ 1 legal staff 2305 Dec 11 09:47 
my_script_1.py
-rwxr-xr-x@ 1 legal staff 1714 Jan 6 2022 
my_script_2.py

<...>

Python env is declared in Pipfile:

$ cat ./Pipfile
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
xlsx2csv = "*"

[dev-packages]
autopep8 = "*"

[requires]
python_version = "3.9"

==> 2. How I currently work with it using “pipenv”

$ cd /path/to/project/dir

$ pipenv shell
$ ./my_script_1.py
$ ./my_script_2.py

Is there any way how to do it simpler using flox or nix?
It would be great if I can get them just as commands/symlinks/etc “my_script_1” and “my_script_2” available in my shell from anywhere, without a need to cd into the dir.

Thank you!

3

Hi @mikhail.zholobov!

flox could be used in few ways depending on your level of nix knowledge. My favorite way (and also a way that requires the least amount of Nix knowledge, if any) is to use flox to provide all the packages you would usually need to use your system package manager.

Lets look at your example and try to create a flox environment that will provide correct python version as well as all the system level tools.

1. Search for your python version

As I see from your Pipfile you are using python 3.9, lets look which version is available in flox.

$ nix search python39
python39 - A high-level dynamically-typed programming language
  stable.nixpkgs-flox.python39@3.9.14
  stable.nixpkgs-flox.python39@3.9.15
  stable.nixpkgs-flox.python39@3.9.16
  staging.nixpkgs-flox.python39@3.9.14
  staging.nixpkgs-flox.python39@3.9.15
  staging.nixpkgs-flox.python39@3.9.16
  unstable.nixpkgs-flox.python39@3.9.14
  unstable.nixpkgs-flox.python39@3.9.15
  unstable.nixpkgs-flox.python39@3.9.16

python39Full - A high-level dynamically-typed programming language
  stable.nixpkgs-flox.python39Full@3.9.14
  stable.nixpkgs-flox.python39Full@3.9.15
  stable.nixpkgs-flox.python39Full@3.9.16
  staging.nixpkgs-flox.python39Full@3.9.14
  staging.nixpkgs-flox.python39Full@3.9.15
  staging.nixpkgs-flox.python39Full@3.9.16
  unstable.nixpkgs-flox.python39Full@3.9.14
  unstable.nixpkgs-flox.python39Full@3.9.15
  unstable.nixpkgs-flox.python39Full@3.9.16

The difference between python39 and python39Full is that the Full package comes with all extra dependencies that you usually want python to come with (eg. openssl, curses, …).

At the time of writing the latest release of python 3.9 (from python.org) is 3.9.16 which is what is also available in flox. There are also other older versions available.

2. Create flox environment with python

$ flox install -e py39-pipenv stable.nixpkgs-flox.python39Full
created generation 1

We just created a flox environment called py39-env with python 3.9.

If we don’t specify the exact version flox will always choose the latest one.

3. Adding pipenv to the environment

$ flox search pipenv
pipenv - Python Development Workflow for Humans
  stable.nixpkgs-flox.pipenv@2022.10.12
  stable.nixpkgs-flox.pipenv@2022.11.11
  stable.nixpkgs-flox.pipenv@2022.11.25
  staging.nixpkgs-flox.pipenv@2022.10.25
  staging.nixpkgs-flox.pipenv@2022.11.11
  staging.nixpkgs-flox.pipenv@2022.11.25
  unstable.nixpkgs-flox.pipenv@2022.10.25
  unstable.nixpkgs-flox.pipenv@2022.11.11
  unstable.nixpkgs-flox.pipenv@2022.11.25

$ flox install -e py39-pipenv stable.nixpkgs-flox.pipenv
created generation 2

4. Using py39-pipenv environment

$ flox activate -e py39-pipenv

flox [py39-pipenv default] [rok@cercei:~/tmp]$ ls -la
-rw-r--r--  1 rok users   196 Feb  9 12:39 Pipfile
-rw-r--r--  1 rok users  4733 Feb  9 12:37 test.xlsx

flox [py39-pipenv default] [rok@cercei:~/tmp]$ pipenv shell

flox [py39-pipenv default] [rok@cercei:~/tmp]$ xlsx2csv test.xlsx
A1,B1,C1
A2,B2,C2

Final thoughts

So flox can be used to provide the correct version of python, pipenv and any other system package that you might require. After that you activate flox environment and use pipenv as you are used to.

Now this is a short way how to start using flox today without the need to learn any Nix.

There are also other things you can do with it and I would invite you to checkout out the documentation.

And if you have any questions just ask them right here.

1 Like
4

Thank you, @garbas !
That looks interesting, I’ll try it out.
Sorry if it sounds like a dumb question, but is there a way to do the same without calling pipenv shell ?

It would be great if I can get them just as commands/symlinks/etc “my_script_1” and “my_script_2” available in my shell from anywhere, without a need to cd into the dir.

Thank you!

5

I’ll provide two options for the question:

Option 1: Building a custom package and publish it to custom channel.

This option would be the more proper way of solving this issue at the moment.

The problem with this option is that it requires substantial knowledge of Nix.

At flox we are aware of this huge mountain users need to climb to be able to package and reuse something with Nix. As I write this we are working on a solution which would make this much more approachable.

Unless you are already familiar with Nix or you want to dive deep into Nix I would advise against this approach.

Option 2: Use declarative environments.

Please read the tutorial on declarative environments and get familiar with flox edit command.

While this option requires you to write Nix, it is far from the level it is required with Option 1. I think it is possible to understand the Nix code below without too many explanations.

For the purpose of the example I’m going to assume few things:

  • Your script is something simple (no modules etc..). I’ll call this `my_script.py
    import sys
import xlsx2csv
xlsx2csv.Xlsx2csv("test.xlsx", outputencoding="utf-8").convert(sys.stdout)
  • Your project is located in /path/to/project.
    This is a path to a folder where Pipfile and your scripts are located

Next is to use the flox edit command and configure few things in this environment.

$ flox edit -e custom-scripts
<OPENS AN EDITOR>

You should write something like this:

  1 {
  2   # packages in environment
  3   packages.nixpkgs-flox.python39 = {};
  4   packages.nixpkgs-flox.pipenv = {};
  5
  6   # env variable PROJECT which holds an absolute path to your project
  7   environmentVariables.PROJECT = "/home/rok/tmp";
  8
  9   # alias my-script to run my_script.py
 10   shell.aliases.my-script = "pipenv run python $PROJECT/my_script.py";
 11
 12   # shell hook that gets run when you activate the environment
 13   # we want to ensure pipenv install is always run and dependencies are present
 14   shell.hook = ''
 15     # we also pipe the output to somewhere we can later review if needed
 16     pipenv install >>$PROJECT/env.log 2>&1
 17   '';
 18 }

Make sure you save the file, which will update the custom-scripts environment.

To run my-script you can now do:

$ flox activate -e custom-scripts

flox [custom-scripts default] $ my-script
A1,B1,C1
A2,B2,C2

Now there are some drawback to Option 2, but maybe the tradeoff is worth it.

Hope this helps.

6

I’m trying to do something similar to Option 2, but using poetry, and running into difficulties.
In order:

  • I have a pyproject.toml:
[tool.poetry]
name = "flox_poetry"
version = "0.1.0"
description = ""
authors = ["Me"]

[tool.poetry.dependencies]
python = ">=3.8,<3.11"
numpy = "^1.19"

[tool.poetry.group.dev.dependencies]
pre-commit = "^3"
black = "^22"
mypy = "^0.991"
ruff = "^0.0.241"

[tool.poetry.group.test.dependencies]
pytest = "^7"
pytest-cov = "^4"

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.black]
line-length = 88

[tool.ruff]
line-length = 120

# allow unused variables when underscore-prefixed.
dummy-variable-rgx = "^(_+|(_+[a-zA-Z0-9_]*[a-zA-Z0-9]+?))$"

# assume Python 3.8
target-version = "py38"

[tool.mypy]
python_version = "3.8"
exclude = [
   # exclude sub-folder of `tests` from type checking
   "tests/*/*.py",
]

[[tool.mypy.overrides]]
ignore_missing_imports = true

[tool.pytest.ini_options]
addopts = [
  "--import-mode=importlib",
]
markers = [
  "slow: mark test as slow (>=1 minute runtime)",
]
  • I create an environment flox create -e flox_poetry
  • I run flox edit -e flox_poetry such that the contents of the file defining the declarative environment are as follows:
{
  packages = {
    nixpkgs-flox = {
      python3Full = {};
      poetry = {};
    };
  };

  shell.hook = ''
     echo "Flox Environment"
     poetry install
     poetry shell
  '';
}

Running flox activate -e flox_poetry shows that poetry is doing its thing and I end up in a poetry shell within the flox shell, as I expect.
However, python -c "import numpy" complains that it can’t load zlib. Adding zlib = {}; to the packages.nix-flox set doesn’t help though.