I was trying to publish a package with a recent version of flox. I can see that publish is still considered experimental but I am lacking a bit of information from the documentation.
flox publish requires a cache URL. I conjectured that https://cache.nixos.org would do the trick
signing-key is less clear to me
For signing-key I first tried to give the path of my ~/.ssh/id_ecdsa_sk which actually uses Yubikey. That didn’t work so I tried to generate other keys with:
Could you please tell me how to generate a proper signing key and publish my package?
The other thing which I find confusing with the documentation about publishing is the notion of channel.
The documentation says that after publishing a package a catalog is created in the package github repository. Then it shows that we can create a channel, more or less by subscribing to that repository. This breaks a bit my mental model since one package = one repository = one catalog = one channel but a channel is supposed to give access to several packages.
at this point the publish user experience works for us (flox) to make our own packages available but many of the rough edges must be smoothed out before it’s ready for real world usage.
we’ll send some clarifications on key generation shortly
The other thing which I find confusing with the documentation about publishing is the notion of channel.
that’s good feedback about some confusion between channel & catalog. multiple channels is so you can have your own catalog alongside flox’s.
Hi @etorreborre, I saw your blog about flox and loved it!
re the channel query, let me try to explain and perhaps together we can find out where we need to clarify this more clearly in the docs.
Once a package builds successfully, it can be published to a channel. People can subscribe to a channel and select a package from that channel to use. Channels and Catalogs are two sides of the same coin and often the terms are used interchangeably. They are simply a collection of packages that have successfully built and been published. flox has a couple default channels (flox and nixpkgs-flox) as examples but you can create your own as well.
Is the statement in the doc " flox publish will create a detached catalog/<system> branch in the upstream repository (https://github.com/flox-examples/hello-python )." under step 2 on the page Publish custom packages | flox:docs causing the confusion?
Hi @etorreborre, I saw your blog about flox and loved it!
Glad you liked it :-).
Is the statement in the doc " flox publish will create a detached catalog/<system> branch in the upstream repository (https://github.com/flox-examples/hello-python )." under step 2 on the page Publish custom packages | flox:docs causing the confusion?
Yes. I don’t mind keeping the two terms but it would be good to explain how they relate to each other. For example we could say that Catalog = where you publish your packages and Channel = a subscription to a given Catalogue?
My main difficulty right now is how to publish my (n) packages to a unique (1) catalog. I doesn’t seem that this is something that I can control with the current flox publish command. In that case that would force me to create as many channels as the number of packages I published.
thanks for this feedback. I know there have been a lot of changes behind the scenes in the CLI as we are moving to our GA so I will need to chat internally with the engineers whether things are currently in limbo or not. The intention is that you could (and should) be able to this.
Hopefully the following will let you workaround, but it’s all still very rough:
You do need to setup a binary cache. I’m not sure what failure you were seeing with wiki instructions, but happy to try to help if you give more details. Another option is to use s3 as described in Serving a Nix store via S3 - Nix Reference Manual. Then pass the s3 URL as --cache-url and set your s3 creds using environment variables.
I was able to use a key generated with nix key generate-secret --key-name mkenigs > mkenigs-key.sec. Is that failing for you?
The current way to publish many packages in a single catalog is by doing something like Add capacitated telegraf to packageset by mkenigs · Pull Request #1 · etorreborre/floxpkgs · GitHub and then running flox publish in that repo (make sure your branch has an upstream, git push -u). I added a little bit of explanation for why we currently do it that way in the PR, but that’s something we haven’t nailed down, so if you have feedback on how to make that easier to use and understand, we’d love to hear it!
But my problem is the signing at the moment. I created a key like you did but I am getting this error message
Signing binary...
error: you are not privileged to add signatures
ERROR: Could not sign binary with sign-key "/Users/etorreborre/.ssh/etorreborre-key.sec"
I wonder if this is related to this issue: nix sign-paths as unprivileged user? · Issue #1708 · NixOS/nix · GitHub. Should I add myself as a trusted user on Mac? Or is this issue solved in a more recent version of Nix? I am using nix 2.17.0 but that doesn’t mean that this is what flox itself is using.
The current way to publish many packages in a single catalog
Thanks for the PR and the explanation. Unfortunately, as a flox user this raises many more questions which I don’t know how to answer :-)!
catalog, channel, environment and now “packageset”?
you talk about flox.lock. As a flox user I don’t really know what that is. It seems to be locking a bunch of dependencies related to flox itself (like shellHooks) and not to my project (no telegraf in there). I kind of understand though that this is a way to make sure that all the packages I publish more or less refer to the same dependencies when possible (and when it’s not possible what happens?)
then, an entry like {capacitated}: capacitated.telegraf.packages.telegraf seems a bit magical to me. I supposed that {capacited} is something that flox is expecting but how the value capacitated.telegraf.packages.telegraf is build and what each part means is, is a mystery to me
the other thing that is not clear to me is where do I specify a name that would be the name of the catalog for all my published packages? My expectation would be that I can reuse that name in other projects to declare “where” they are published
also the doc for flox publish talks about a cache and a public-cache, the difference being where to pull or where to push binaries. Should I understand that cache is where the binaries for the current packages are pushed and public-cache is where the binaries needed to build the current package are being retrieved?
I am happy to be used as guinea pig if you need someone to try out the publish workflow in real-time, so that you can see what kind of questions I ask myself while doing it.
Most of those are fantastic questions, and they’re questions we need to simplify and address at some point. Unfortunately for now I think some things will remain mysterious because they aren’t really ready to be used. But I’ll try to answer as best I can.
catalog and channel should probably be collapsed into a single concept. The distinction between packageset and catalog/channel is that packageset is essentially source, and catalog/channel contains (references to) built binary artifacts.
flake.lock locks all of your nix dependencies, so in the case of telegraf that would most notably be buildGoModule, which determines your version of go used to build telegraf. The whole packageset and capacitated rigamarole would ensure that the same version of buildGoModule is used for any package in the packageset. In the case of go that’s somewhat unimportant because of static binaries, but it becomes more useful if multiple packages in the packageset both use the same binary or library dependency, e.g. libssl.
The capacitated stuff says to ignore the flake.lock of telegraf and instead provide the buildGoModule used by the packageset. The first telegraf says to use the telegraf flake input defined in flake.nix. Nix flakes can define multiple different types of outputs, one of which is packages. And the final telegraf is the name of the package defined in the telegraf flake. So that corresponds to github:etorreborre/telegraf#packages.{system is handled automatically}.telegraf, but with the capacitated stuff ignoring the lockfile.
I’m not sure if I fully understand that question - do you mean you would expect to be able to say catalog_I_publish_to = github:etorreborre/floxpkgs in your telegraf repo?
the difference is primarily for authentication. E.g. we use s3://... as our cache and supply credentials, but then public-cache is https://cache.flox.dev... and doesn’t require creds.
Thanks for being so willing to try things out! Pinging @ghudgins and @ysndr as they’ll likely be involved when we do more reworking of publish.
I’m not sure if I fully understand that question - do you mean you would expect to be able to say catalog_I_publish_to = github:etorreborre/floxpkgs in your telegraf repo?
Maybe not in the repository. Maybe just at publishing time.
I think that polishing “publish” is super important because it is the escape hatch when something is not available in the standard nix packages (possibly intentionally if this is some kind of private/internal package).
Maybe not in the repository. Maybe just at publishing time.
Yeah that’s something to keep in mind - we currently only support publishing to the repo a package is defined in. That encourages creating a packageset so that all packages are cohesive. But I agree it’s a bit surprising that you can’t specify an alternative channel name to publish to.
Agreed
We definitely will be revisiting flox publish in light of the changes we are currently making to the CLI. The current estimate is early Q1 but if we can get through things faster, hopefully we can get to it sooner!
In the meantime hopefully Matthew’s notes will give you a workaround.
