Using conda as a backend? Is it worth supporting?

[tkf]

Since people started asking about (ana)conda in random places, I open this as an issue here.

[tkf]

Branching off from @asmeurer's comment at tkf/emacs-jedi#140 (comment)

No, conda environments are not virtualenvs. [...] Conda environments are basically done the "right" way:

Hmm... That is interesting. Do you know a good document to start with understanding conda architecture? I find explanation saying that conda is a wrapper on top of virtualenv but I guess it's not true?

I have a feeling like conda treats C extension better. I haven't started using it but actually want to start using it since I want better handling of C extension and its shared library dependencies.

But virtualenv should still be a supported by python-environment.el. Reading http://conda.pydata.org/docs/commands/install.html it looks like that it is possible to implement a compatible layer in python-environment.el. But still I don't get why you'd want to do that. In the end, python-environment.el is just for installing Jedi and EPC. You will never user its environment manually and you should not care at all what is going on behind the scene.

[tkf]

@spearsem You asked a similar question here tkf/emacs-jedi#140 (comment) so feel free to comment or start some discussion.

[spearsem]

I'll defer to @asmeurer but I'm more than happy to help implement anything if this results in a proposal.

I agree with the spirit of your comment, "You will never user its environment manually and you should not care at all what is going on behind the scene." But with the advent of conda as a superior package management tool in the Python ecosystem, it might becoming more popular to avoid installing / using virtualenv at all. That was my own case when I made my initial comment -- I want to manage packages using only conda, but had to install virtualenv for jedi (or allow jedi to install it) which I'd prefer not to do.

If the backend just checked whether conda is installed and prompted the user for the option, that might be fine. Or maybe some kind of flag can be set. Like an environment variable PYTHON_PACKAGE_MANAGER?

[asmeurer]

Yes, conda does handle C extensions better (it also helps that its a binary package manager, as opposed to source). That was basically the initial motivation for conda, because tools like pip are not well suited to C extensions, which are used heavily in the scientific community. Also, conda is really Python agnostic. You can use it to manage anything, as long as you can build a relocatable package for it.

To support conda, I guess what is needed is to allow Jedi to look in multiple locations for the source, so that I can just point it to my root conda environment where I tend to have everything installed. I actually can probably hack this up already by fiddling with PYTHONPATH or some .pth file in the jedi env (I'll have to think about it).

[tkf]

@spearsem

I want to manage packages using only conda, but had to install virtualenv for jedi (or allow jedi to install it) which I'd prefer not to do.

You can install virtualenv via conda, right? I guess you still need virtualenv for tox so I guess you will have it as long as you are programming in Python. But I understand the concern about reducing command you use.

Like an environment variable PYTHON_PACKAGE_MANAGER?

Well, it's just an Emacs package so it will be just (setq ...).

@asmeurer

tools like pip are not well suited to C extensions, which are used heavily in the scientific community

Why? I install scientific tools all the time using pip. Most of the time these tools provide a way to tell compile options etc. so I never felt like pip reject scientific ecosystems.

conda is really Python agnostic

This is interesting. I want all programming language to have something like python-environment.el so that you never run install command for external dependencies. But writing hundreds of LANGAGE-environment.el sounds just stupid. Maybe we can just have conda.el and forget about all these LANGAGE-environment.el. That would be better.

allow Jedi to look in multiple locations for the source, so that I can just point it to my root conda environment where I tend to have everything installed

This is another problem which must be treated in tkf/emacs-jedi#154

You are talking about how to tell emacs-jeid about conda environments. This is not about using conda as a python-environmen.el backend (i.e., using conda to install stuff). I think it is already supported in emacs-jedi. (@asmeurer you may already noticed about the other issue, but I am writing this anyway for other people reading this issue).

[tkf]

@asmeurer Re: pip, maybe you are talking about binary support missing in pip? Well then, yes, that is what is missing in pip.

[asmeurer]

Sure, but there are also some packages that rely on C libraries that are not themselves Python libraries.

And before you say "just apt-get install it" remember that more than 50% of the users are on Windows (that's another thing about conda that is missing in many alternatives: it is cross-platform).

[tkf]

some packages that rely on C libraries that are not themselves Python libraries.

So conda solves this case too? Cool!

And before you say "just apt-get install it"

No I say "build from tar" :-) Doing science in Windows? How unfortunate...

[tkf]

Anyway, I am convinced conda is better than virtualenv. I am sure I'll use it personally. Thank you guys.

But I won't write conda.el anytime soon since:

• virtualenv is still standard outside of science community. I am not convinced until tox rely on conda instead of virtualenv.
• virtualenv just works. you can install it using conda.
• only emacs-jedi is using it and for the case of emacs-jedi, conda is not needed at all.

[asmeurer]

As I noted, if you add the ability to Jedi or emacs-jedi to support multiple site-packages (it sounds like you were looking at doing something like that anyway?), then that will basically fix the issue.

Or, you can just install anaconda into the Jedi environment (I don't know if I pointed it out, but conda environments are efficient because they just use hard-linking, so you can create many of them with many packages without using a ton of disk space). Or just point jedi to your root environment (i.e., don't use any kind of virtual environment at all).

[tkf]

@asmeurer You are still confusing.

When you say "emacs-jedi to supporting multiple site-packages", there are two possible interpretations.

1. Support installation of emacs-jedi with multiple Python versions. It is treated in this issue.
2. Support "analysing" modules in multiple site-packages, including conda environments. As I said several times, it is already supported in emacs-jedi (not perfect against C exts, of course). However, as I haven't started conda, I cannot give you a concrete example. It is treated in How to tell emacs-jedi about conda environments emacs-jedi#154 not here.