Installing greta: TensorFlow and TensorFlow Probability installed but not detected

essicolo · February 5, 2019, 1:31am

In December, I had a hard time installing greta, but once installed ended up enjoying it a lot. I had to reinstall Ubuntu and tried all sorts of things to install greta, but still get the same error. The subject seems to be widely covered on GitHub, but unfortunately without a working solution, at least on my machine.

devtools::install_github("greta-dev/greta")
library("greta")
greta::install_tensorflow(method = "conda")
reticulate::conda_install("r-tensorflow", "tensorflow-probability", pip = TRUE)

int <- normal(0, 5)
Erreur : 

greta requires TensorFlow (>=1.10.0) and Tensorflow Probability (>=0.5.0), but TensorFlow isn't installed and TensorFlow Probability isn't installed. Use:

    install_tensorflow(method = "conda")
    reticulate::conda_install("r-tensorflow", "tensorflow-probability", pip = TRUE)

to install the latest version.

Voltemand · February 5, 2019, 7:41am

Hi essicolo,

Thanks for question, it’s a pity you’re having these issues.

Could you try to run the following in R to give us a better sense for the problem:

tensorflow::tf_config()
reticulate::conda_list()
reticulate::py_module_available("tensorflow") 
reticulate::py_module_available("tensorflow_probability")

Thanks!

essicolo · February 5, 2019, 3:38pm

HI,

Thanks. Indeed, reticulate can’t find tensorflow.

> tensorflow::tf_config()
Installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /home/xyz/anaconda3/envs/r-tensorflow/bin/python3.6
 /usr/bin/python2.7
 /usr/bin/python3.6
 /home/xyz/anaconda3/bin/python3.7

You can install TensorFlow using the install_tensorflow() function.
 
> reticulate::conda_list()
          name                                            python
1    anaconda3                   /home/xyz/anaconda3/bin/python
2 r-tensorflow /home/xyz/anaconda3/envs/r-tensorflow/bin/python
> reticulate::py_module_available("tensorflow")
[1] FALSE
> reticulate::py_module_available("tensorflow_probability")
[1] FALSE

There were unmet dependencies during the installation of tensorflow-probability. I tried both to downgrade numpy and to install tensorflow-probability from anaconda.org, but reticulate keeps ignoring it.

> reticulate::conda_install("r-tensorflow", "tensorflow-probability", pip = TRUE)
Collecting tensorflow-probability
  Downloading https://files.pythonhosted.org/packages/a1/ca/6f213618b5f7d0bf6139e6ec928d412a5ca14e4776adfd41a59c74a34021/tensorflow_probability-0.5.0-py2.py3-none-any.whl (680kB)
Collecting numpy>=1.13.3 (from tensorflow-probability)
  Downloading https://files.pythonhosted.org/packages/f5/bf/4981bcbee43934f0adb8f764a1e70ab0ee5a448f6505bd04a87a2fda2a8b/numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl (17.3MB)
Collecting six>=1.10.0 (from tensorflow-probability)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
tensorflow 1.10.0 has requirement numpy<=1.14.5,>=1.13.3, but you'll have numpy 1.16.1 which is incompatible.
tensorflow 1.10.0 has requirement setuptools<=39.1.0, but you'll have setuptools 40.7.3 which is incompatible.
Installing collected packages: numpy, six, tensorflow-probability
Successfully installed numpy-1.16.1 six-1.12.0 tensorflow-probability-0.5.0

The conda list shows tensorflow and tensorflow-probability.

(r-tensorflow) xyz@pop-os:~$ conda list
# packages in environment at /home/xyz/anaconda3/envs/r-tensorflow:
#
# Name                    Version                   Build  Channel
absl-py                   0.7.0                    pypi_0    pypi
asn1crypto                0.24.0                py36_1003    conda-forge
astor                     0.7.1                    pypi_0    pypi
blas                      1.1                    openblas    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
cffi                      1.11.5          py36h9745a5d_1001    conda-forge
chardet                   3.0.4                 py36_1003    conda-forge
cryptography              2.4.2            py36h1ba5d50_0  
freetype                  2.9.1             h94bbf69_1005    conda-forge
gast                      0.2.2                    pypi_0    pypi
grpcio                    1.18.0                   pypi_0    pypi
h5py                      2.9.0           nompi_py36hf008753_1102    conda-forge
hdf5                      1.10.4          nompi_h11e915b_1105    conda-forge
idna                      2.8                   py36_1000    conda-forge
jpeg                      9c                h14c3975_1001    conda-forge
keras                     2.2.4                    pypi_0    pypi
keras-applications        1.0.7                    pypi_0    pypi
keras-preprocessing       1.0.8                    pypi_0    pypi
libffi                    3.2.1             hf484d3e_1005    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libpng                    1.6.36            h84994c4_1000    conda-forge
libstdcxx-ng              7.3.0                hdf63c60_0    conda-forge
libtiff                   4.0.10            h648cc4a_1001    conda-forge
markdown                  3.0.1                    pypi_0    pypi
ncurses                   6.1               hf484d3e_1002    conda-forge
numpy                     1.14.5                   pypi_0    pypi
olefile                   0.46                       py_0    conda-forge
openblas                  0.3.3             h9ac9557_1001    conda-forge
openssl                   1.1.1a            h14c3975_1000    conda-forge
pillow                    5.4.1           py36h00a061d_1000    conda-forge
pip                       19.0.1                   py36_0    conda-forge
protobuf                  3.6.1                    pypi_0    pypi
pycparser                 2.19                       py_0    conda-forge
pyopenssl                 19.0.0                   py36_0    conda-forge
pysocks                   1.6.8                 py36_1002    conda-forge
python                    3.6.8                h0371630_0  
pyyaml                    3.13            py36h14c3975_1001    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
requests                  2.21.0                py36_1000    conda-forge
scipy                     1.2.0           py36_blas_openblash1522bff_1201  [blas_openblas]  conda-forge
setuptools                39.1.0                   pypi_0    pypi
six                       1.12.0                   pypi_0    pypi
sqlite                    3.26.0            h67949de_1000    conda-forge
tensorboard               1.10.0                   pypi_0    pypi
tensorflow                1.10.0                   pypi_0    pypi
tensorflow-hub            0.2.0                    pypi_0    pypi
tensorflow-probability    0.5.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tk                        8.6.9             h84994c4_1000    conda-forge
urllib3                   1.24.1                py36_1000    conda-forge
werkzeug                  0.14.1                   pypi_0    pypi
wheel                     0.32.3                   pypi_0    pypi
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge

Voltemand · February 6, 2019, 4:13am

Thanks for that.

Can you please try:

Fix install manually via shell (Ubuntu):

source activate r-tensorflow
pip install tensorflow==<CORRECT VERSION>
pip install tensorflow-probablity==<CORRECT VERSION>
reboot

Taken from here: https://github.com/greta-dev/greta/issues/254 (with thanks to ettepo)

Thanks!

essicolo · February 6, 2019, 6:09pm

I have found the trick. The origin of the error was not greta, nor reticulate, but Anaconda. When I did pip install tensorflow while the r-tensorflow environment was activated, tensorflow was installed in .local/lib/python36…, not in the r-tensorflow environment. That’s why it was absent from conda list (although it was there in the output of a previous post, I probably did a conda install instead). I had to do the following.

conda create --name r-tensorflow python==3.6
anaconda3/envs/r-tensorflow/bin/pip install tensorflow
anaconda3/envs/r-tensorflow/bin/pip install tensorflow-probability

And… bingo!

Thanks for the help!

nick · February 11, 2019, 12:53am

I’m glad you got it working!

Yeah, this is turning out to be a complete pain. I think something’s changed in how anaconda does things, and the tensorflow and reticulate packages (which greta relies on for these things) don’t seem to work as they used to.

It’s pretty hard to keep up with all these little changes stuff, and I have yet to find a robust solution (suggestions very welcome!).

I arrived at a very similar solution a couple of days ago when setting up a new laptop. I think the following is more general and more in keeping with conda’s intended usage, but I’m no expert:

conda create r-tensorflow
conda activate r-tensorflow
conda install tensorflow
pip install tensorflow-probability
conda deactivate

It might be easiest to wrap these up in R system calls and provide them in a new greta::install_tensorflow(), though I’d rather investigate why the reticulate approach doesn’t work, and post a fix there if possible.

Voltemand · February 11, 2019, 10:44am

This is a pain!

The offending file in question is conda.R in reticulate. In particular the crucial conda_install function here in that file seems very strange to me.

On OSX it sources (i.e. uses the unix source command) a bin/activate file in the specified conda environment, a method of environment activation that I can’t find recommended or even documented anywhere.

Given that this is clearly a huge pain point for users I think the idea of creating a greta::install_tensorflow() function is a good one, though it would have to take into account mac vs windows, recent updates to conda etc.

ajf · February 11, 2019, 4:35pm

Got an encouraging alert over the weekend that TFP is on conda-forge now thanks to @sodre: https://github.com/conda-forge/tensorflow-probability-feedstock

I have not tested it yet, but this could simplify life.
See here for more details. https://github.com/tensorflow/probability/issues/211

dirmeier · February 12, 2019, 9:04am

Just to add my two cents (if anyone cares): I’d recommend to install from conda-forge in any case, since it has unit tests, CI and makes installation of tensorflow-gpu quite easy. To make sure that you get the TFP version that works with greta 0.3.0, maybe add the version number, too:

conda install tensorflow-probability==0.5.0

nick · February 13, 2019, 6:26am

Oh wow, I’ve been waiting for it to get to conda forge and it looks like it went up 2 days ago!

Yes, totally agree, that’s much better. My installation instructions above would then be:

conda create r-tensorflow
conda activate r-tensorflow
conda install tensorflow tensorflow-probability==0.5.0
conda deactivate

It should now be easier to install from within R (when we can resolve that reticulate issue), as it will be:

tensorflow::install_tensorflow(method = "conda",
                               extra_packages = "tensorflow-probability")

nick · February 13, 2019, 6:31am

I should really read all the new messages in the thread before I comment 🤦 Thanks for the heads up @ajf!