I’d like to use gdal_calc to do geographic processing.
With plain-old nix-shell I can do this:
nix-shell -p "python3.withPackages(p: [p.numpy p.gdal])" --run "gdal_calc.py --help"`
or create an equivalent shell.nix file like this:
{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
buildInputs = [
(pkgs.python3.withPackages(p: [p.numpy p.gdal]))
];
}
How can I achieve the same with flox?
Thanks.
Hi, this would be one way to do it:
first set up an environment with:
flox install -e gdalEnv python3 nixpkgs#python3Packages.numpy nixpkgs#python3Packages.gdal
then run a command without entering that environment:
flox activate -e gdalEnv -- python gdal_calc.py
Effectively your environment because your shell.nix file, but you can activate it in any directory.
I think there’s a better way to do this which I will follow up on if I find it.
Hi @paulovieira - we are currently working on an “inline packaging” feature that will make it super easy to create arbitrary python “withPackages” environments, but in the meantime you can create your own “gdalPython” package that can be installed to an environment like any other. This is probably a good workflow to be aware of in any case and not to worry - with flox this is not nearly as hard as it sounds!
I’ll start by first describing how to use a custom package, and then we’ll work backwards to explain how to create one.
I’ve taken the liberty to create a gdalPython package that meets your specifications in the github:flox-examples/floxpkgs repository, and here’s how you can use it:
brantley@Michaels-MBP flox-examples % flox subscribe flox-examples github:flox-examples/floxpkgs
brantley@Michaels-MBP flox-examples % flox install -e gdal flox-examples#gdalPython
created generation 1
brantley@Michaels-MBP flox-internal % flox activate -e gdal
flox [gdal default] brantley@Michaels-MBP flox-internal %
From this shell you can then use any of the gdal programs, or a version of python equipped with all the modules you need:
flox [gdal default] brantley@Michaels-MBP flox-internal % gdalpython
Python 3.10.8 (main, Nov 13 2022, 14:51:22) [Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> from osgeo import gdal
>>>
flox [gdal default] brantley@Michaels-MBP flox-internal % gdal_calc.py --help |& head -3
usage: gdal_calc.py [--help] --calc [expression ...] [-A [filename ...]]
[--A_band [n ...]] --outfile filename
[--NoDataValue value] [--hideNoData] [--type datatype]
flox [gdal default] brantley@Michaels-MBP flox-internal %
Easy, eh? But how do you create your own custom package in your own package collection?
floxpkgs collectionWe’re thinking about better ways to do this in future, but at present the way to create your own floxpkgs collection is to publish something to it. It doesn’t matter what that is, so in this case I’ll publish the gdalPython package from above to a new limeytexan/floxpkgs repository:
flox [gdal default] brantley@Michaels-MBP flox-internal % flox publish flox-examples#gdalPython
package name: gdalPython
+ /nix/store/qvfslp3zix6zv2bl56lgbw9dld9zpw8q-flox-0.0.7-r76/libexec/flox/gh repo create --public git@github.com:limeytexan/floxpkgs --template https://github.com/flox/floxpkgs-template.git
✓ Created repository limeytexan/floxpkgs on GitHub
channel repository: git@github.com:limeytexan/floxpkgs
HINT: avoid having to answer these questions next time with:
$ flox publish -A gdalPython --build-repo https://github.com/flox-examples/floxpkgs --channel-repo git@github.com:limeytexan/floxpkgs
Cloning git@github.com:limeytexan/floxpkgs ...
Cloning into '/var/folders/_s/s7vwf8td57xbnh5h2styht2r0000gn/T/tmp.OhiFhJAg71'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 7 (delta 0), reused 5 (delta 0), pack-reused 0
Receiving objects: 100% (7/7), done.
Building gdalPython ...
publishing render to catalog ...
flox publish completed
[main 65b3067] published aarch64-darwin/stable/gdalPython/3.10.8-env.json
1 file changed, 111 insertions(+)
create mode 100644 catalog/aarch64-darwin/stable/gdalPython/3.10.8-env.json
Enumerating objects: 9, done.
Counting objects: 100% (9/9), done.
Delta compression using up to 10 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (7/7), 1.93 KiB | 1.93 MiB/s, done.
Total 7 (delta 0), reused 0 (delta 0), pack-reused 0
To github.com:limeytexan/floxpkgs.git
7f90744..65b3067 main -> main
flox [gdal default] brantley@Michaels-MBP flox-internal %
Let’s step through all that happened above:
flox publish asks for a) the URL from which the package can be built, and b) the URL where it should publish the build results. In this case we can build gdalPython from https://github.com/flox-examples/floxpkgs and I’m wanting to publish to my own floxpkgs repository, which it will automatically create for you in the process (from the floxpkgs-template).<enter> to skip those fields for now.floxpkgs repository as provided above.If you do all that yourself (creating the pauloviera/floxpkgs repository in the process), you will now have a new way to use that package!
flox subscribe pauloviera github:pauloviera/floxpkgs
flox install -e gdal pauloviera.gdalPython
But that’s still not quite the whole story because you need to be able to update that package over time, so let’s add that gdalPython package to your collection.
In this case you can just copy the example I created, but it’s a similar (and similarly straightforward) process to create packages for other languages. This should do the trick:
git clone git@github.com:pauloviera/floxpkgs
cd floxpkgs
mkdir pkgs/gdalPython
wget -O pkgs/gdalPython/default.nix https://raw.githubusercontent.com/flox-examples/floxpkgs/master/pkgs/gdalPython/default.nix
git add pkgs/gdalPython/default.nix
git commit -am "gdalPython: init"
git push
… and following those steps you will be able to install or publish directly from your floxpkgs clone. Try invoking flox publish from your clone to see what I mean.
If you’ve made it this far it wouldn’t surprise me if this wasn’t still a bit confusing, and in particular you may be wondering about the difference between pauloviera#gdalPython and pauloviera.gdalPython. This is an important distinction, let me try to clear that up:
pauloviera#gdalPython is the flake URL for building the gdalPython package in github:pauloviera/floxpkgs using the current version of the (stable) nixpkgs collection. When you reference this you are asking flox to build the package for you new every time. This can (and does) fail over time as nixpkgs moves out from under you, and can result in delays as you wait for builds to take place.pauloviera.gdalPython (or to be more accurate, stable.pauloviera.gdalPython) is the floxpkgs triple providing all the information necessary to build or download the latest known-good build of that package. When you reference this you are asking flox to go back through the history of all successful builds and then repeat exactly what worked last time, using whatever version of the (stable) nixpkgs collection was active at the time. This will not fail, and can be very fast when you combine this with saving build artifacts in a binary cache.Which one should you use? At this point while this is all very manual it depends entirely on your use case, but in future we will provide automation to continuously build and update your package collections over time, making the second of the options above generally faster, more convenient and more reliable than the first.
Thanks for asking about this example - it provided the opportunity to cover a ton of flox features and workflow, and hopefully will be of use to you and others!
Thanks Michael for the long explanation, it is also helpful for me.
I’m starting to understand how you “floxify” a repository:
pkgs directory, defined by their nix filepackages rather than pkgs. Could we be more consistent here?)Thank you much for the helpful responses. I haven’t yet had opportunity to try it but I’m looking forward to it. I think this post could be used as a tutorial.