Python bindings not loaded

#1

I am trying to install greta on a windows10 machine (sigh), I have tried:

  1. installing from CRAN via install.packages(“greta”), and installing tensorflow using
install_tensorflow(
    method = "conda",
    version = "1.14.0",
    extra_packages = "tensorflow-probability"
  )

The issue is that I end up with tensorflow_2.0.0 and not 1.14.0 as expected by the package, and so after re-starting and re-loading the package, I (still) get error messages about missing requirements …

  1. Following this post: Can't install Tensorflow 1.14 with greta and this one Installing Greta: Robust instructions as of March 31, 2019, I have turned to the dev version:
devtools::install_github("greta-dev/greta")

But I get Python bindings issues:

   Error: package or namespace load failed for 'greta':
     .onLoad failed in loadNamespace() for 'greta', details:
      call: initialize_python(required_module, use_environment)
      error: Installation of Python not found, Python bindings not loaded.

Then I added the python path to the Path environment variable following: https://superuser.com/questions/949560/how-do-i-set-system-environment-variables-in-windows-10, but to no avail I still get Python binding errors … Should I just wait a bit for the greta update with tensorflow_2.0 to trickle down or do someone else out there see an option?

#2

I think you just need extra_packages = "tensorflow-probability==0.7.0" in the install call.

A new version of TFP was released that depends on TF2, and the previous install instructions weren’t robust to that. Instructions are fixed in the GitHub version, but it’s not on CRAN yet. Let me know if that works!

#3

Thanks @nick for your super-fast answer.

What I tried:

  1. remove.package(“greta”)
  2. install.packages(“greta”)
  3. install_tensorflow(
    method = “conda”,
    version = “1.14.0”,
    extra_packages = “tensorflow-probability==0.7.0”
    )

I still get error messages after restarting R session when loading greta (“This version of greta requires TensorFlow v1.14.0 and TensorFlow Probability v0.7.0, but TensorFlow isn’t installed and TensorFlow Probability isn’t installed.”), and a call to sessionInfo() reveal that tensorflow_2.0 is still the package being installed and loaded.

#4

I recently reinstalled Anaconda 3. I had success installing and running Greta by:

  1. Creating a seperate conda environment by cloning the base then installing tensorflow 1.14 with conda and tensorflow probability 0.7.0 with pypi.
  2. Installing tensorflow and greta via cran.
  3. When running greta, use reticulate:use_condaenv
#5

Thanks @pang for your reply. This seems a bit too involved for my python / conda skills so I guess that I’ll just wait for the next greta CRAN release.

#6

I would try:

  1. make sure you have the latest CRAN versions of reticulate and tensorflow

  2. running: reticulate::conda_remove("r-tensorflow") and reticulate::conda_remove("r-reticulate") to clear out the existing envs

  3. restarting R and clearing the environment of all files including hidden ones

  4. running this again:

greta::install_tensorflow(
  method = “conda”,
  version = “1.14.0”,
  extra_packages = “tensorflow-probability==0.7.0”
)

  1. restarting R and clearing the environment of all files including hidden ones, again

  2. reloading greta

Even though the install_tensorflow command restarts the R session itself, it doesn’t seem to reattach the correct python (and therefore see the right conda environment). Older versions of reticulate and tensorflow seems to have trouble installing the right versions of tensorflow. Though I think the main thing if forcibly deleting the old environment with TF2 installed.

If that doesn’t work, please paste the results of sessionInfo(), reticulate::py_config() and tensorflow::tf_config() here. This should be easy to fix, no need to wait until TF2-compatible greta is released!

#7

Thanks for the advice.

I followed the steps you outlined andfelt like it was doing the trick, running the instal_tensorflow command prompted messages such as: “Collecting tensorflow==1.14.0” which made me very optimistic.

But after the installation finished and after restarting R and cleaning up (using rm(list=ls())), I still get the error message that greta requires TensorFlow 1.14.0.

Here are the outputs from sessionInfo():

R version 3.6.1 (2019-07-05)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 18362)

Matrix products: default

locale:
[1] LC_COLLATE=German_Germany.1252  LC_CTYPE=German_Germany.1252    LC_MONETARY=German_Germany.1252
[4] LC_NUMERIC=C                    LC_TIME=German_Germany.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] greta_0.3.1

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.2        rstudioapi_0.10   whisker_0.4       magrittr_1.5      hms_0.5.1         progress_1.2.2   
 [7] lattice_0.20-38   R6_2.4.0          rlang_0.4.0       globals_0.12.4    tools_3.6.1       parallel_3.6.1   
[13] grid_3.6.1        coda_0.19-3       tfruns_1.4        digest_0.6.22     assertthat_0.2.1  tibble_2.1.3     
[19] crayon_1.3.4      tensorflow_2.0.0  Matrix_1.2-17     base64enc_0.1-3   vctrs_0.2.0       codetools_0.2-16 
[25] zeallot_0.1.0     compiler_3.6.1    pillar_1.4.2      backports_1.1.5   prettyunits_1.0.2 reticulate_1.13  
[31] jsonlite_1.6      future_1.14.0     listenv_0.7.0     pkgconfig_2.0.3

reticulate::py_config()

python:         C:\PROGRA~3\ANACON~1\python.exe
libpython:      C:/PROGRA~3/ANACON~1/python37.dll
pythonhome:     C:\PROGRA~3\ANACON~1
version:        3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]
Architecture:   64bit
numpy:          C:\PROGRA~3\ANACON~1\lib\site-packages\numpy
numpy_version:  1.16.4
tensorflow:     [NOT FOUND]

and tensorflow::tf_config()

Installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 C:\ProgramData\Anaconda3\python.exe

You can install TensorFlow using the install_tensorflow() function.
#8

I think you need to do rm(list=ls(all.names = TRUE)) to clear the hidden files. Alternatively, in RStudio there’s a broom icon and option to clear hidden files. Or you can set R to clear the environment and not reload it when you restart a session - usethis::use_blank_slate() will set that up for you. Could you rerun the instructions with one of these to see if that’s the issue?

It’s not using a conda environment as the loaded python, which is strange. Usually a conda environment called r-reticulate is the windows/conda default. This could be due to not clearing out the environment fully, meaning it isn’t switching over to the new environment it created.

When you rerun (and if it doesn’t work), could you paste the messages from installation of tensorflow? That should say which python outstation it’s installing TF1.14 into.

#9

Thanks for your patience, still not working after following the steps you outline and using the Rstudio broom icon to clean all files including hidden ones.

Here are the installation message from the greta::install_tensorflow call:

Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done


==> WARNING: A newer version of conda exists. <==
  current version: 4.7.10
  latest version: 4.7.12

Please update conda by running

    $ conda update -n base -c defaults conda



## Package Plan ##

  environment location: C:\Users\Hertzog\Documents\.conda\envs\r-reticulate

  added / updated specs:
    - python=3.6


The following NEW packages will be INSTALLED:

  certifi            pkgs/main/win-64::certifi-2019.9.11-py36_0
  pip                pkgs/main/win-64::pip-19.3.1-py36_0
  python             pkgs/main/win-64::python-3.6.9-h5500b2f_0
  setuptools         pkgs/main/win-64::setuptools-41.4.0-py36_0
  sqlite             pkgs/main/win-64::sqlite-3.30.1-he774522_0
  vc                 pkgs/main/win-64::vc-14.1-h0510ff6_4
  vs2015_runtime     pkgs/main/win-64::vs2015_runtime-14.16.27012-hf0eaf9b_0
  wheel              pkgs/main/win-64::wheel-0.33.6-py36_0
  wincertstore       pkgs/main/win-64::wincertstore-0.2-py36h7fe50ca_0


Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
#
# To activate this environment, use
#
#     $ conda activate r-reticulate
#
# To deactivate an active environment, use
#
#     $ conda deactivate

Collecting tensorflow==1.14.0
  Using cached https://files.pythonhosted.org/packages/bf/4a/5c86ed8b245aa48f9f819b13a0a9039e9126ba19fdd0c7e0b8026c12315a/tensorflow-1.14.0-cp36-cp36m-win_amd64.whl
Collecting tensorflow-probability==0.7.0
  Using cached https://files.pythonhosted.org/packages/3e/3a/c10b6c22320531c774402ac7186d1b673374e2a9d12502cbc8d811e4601c/tensorflow_probability-0.7.0-py2.py3-none-any.whl
Collecting wheel>=0.26
  Using cached https://files.pythonhosted.org/packages/00/83/b4a77d044e78ad1a45610eb88f745be2fd2c6d658f9798a15e384b7d57c9/wheel-0.33.6-py2.py3-none-any.whl
Collecting numpy<2.0,>=1.14.5
  Using cached https://files.pythonhosted.org/packages/55/7a/f32b39164262765b069b0fe3ec5d4b47580c9c60f7bd3588b58ba8e93a4c/numpy-1.17.3-cp36-cp36m-win_amd64.whl
Collecting six>=1.10.0
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting google-pasta>=0.1.6
  Using cached https://files.pythonhosted.org/packages/d0/33/376510eb8d6246f3c30545f416b2263eee461e40940c2a4413c711bdf62d/google_pasta-0.1.7-py3-none-any.whl
Collecting keras-preprocessing>=1.0.5
  Using cached https://files.pythonhosted.org/packages/28/6a/8c1f62c37212d9fc441a7e26736df51ce6f0e38455816445471f10da4f0a/Keras_Preprocessing-1.1.0-py2.py3-none-any.whl
Collecting tensorboard<1.15.0,>=1.14.0
  Using cached https://files.pythonhosted.org/packages/91/2d/2ed263449a078cd9c8a9ba50ebd50123adf1f8cfbea1492f9084169b89d9/tensorboard-1.14.0-py3-none-any.whl
Processing c:\users\hertzog\appdata\local\pip\cache\wheels\a7\15\a0\0a0561549ad11cdc1bc8fa1191a353efd30facf6bfb507aefc\absl_py-0.8.1-cp36-none-any.whl
Collecting keras-applications>=1.0.6
  Using cached https://files.pythonhosted.org/packages/71/e3/19762fdfc62877ae9102edf6342d71b28fbfd9dea3d2f96a882ce099b03f/Keras_Applications-1.0.8-py3-none-any.whl
Processing c:\users\hertzog\appdata\local\pip\cache\wheels\7c\06\54\bc84598ba1daf8f970247f550b175aaaee85f68b4b0c5ab2c6\termcolor-1.1.0-cp36-none-any.whl
Processing c:\users\hertzog\appdata\local\pip\cache\wheels\59\38\c6\234dc39b4f6951a0768fbc02d5b7207137a5b1d9094f0d54bf\gast-0.3.2-cp36-none-any.whl
Collecting astor>=0.6.0
  Using cached https://files.pythonhosted.org/packages/d1/4f/950dfae467b384fc96bc6469de25d832534f6b4441033c39f914efd13418/astor-0.8.0-py2.py3-none-any.whl
Collecting grpcio>=1.8.6
  Using cached https://files.pythonhosted.org/packages/70/34/b04fa7c1b1bd5e00e2cceea68aa21fa871349405b168394d2218baaa2c13/grpcio-1.24.3-cp36-cp36m-win_amd64.whl
Collecting tensorflow-estimator<1.15.0rc0,>=1.14.0rc0
  Using cached https://files.pythonhosted.org/packages/3c/d5/21860a5b11caf0678fbc8319341b0ae21a07156911132e0e71bffed0510d/tensorflow_estimator-1.14.0-py2.py3-none-any.whl
Processing c:\users\hertzog\appdata\local\pip\cache\wheels\d7\de\2e\efa132238792efb6459a96e85916ef8597fcb3d2ae51590dfd\wrapt-1.11.2-cp36-none-any.whl
Collecting protobuf>=3.6.1
  Using cached https://files.pythonhosted.org/packages/2d/73/4a14606fa26f186e23015bc974f9010e2bbf1607f372e3bd5e82d2a62f1b/protobuf-3.10.0-cp36-cp36m-win_amd64.whl
Collecting decorator
  Using cached https://files.pythonhosted.org/packages/8f/b7/f329cfdc75f3d28d12c65980e4469e2fa373f1953f5df6e370e84ea2e875/decorator-4.4.1-py2.py3-none-any.whl
Collecting cloudpickle>=0.6.1
  Using cached https://files.pythonhosted.org/packages/c1/49/334e279caa3231255725c8e860fa93e72083567625573421db8875846c14/cloudpickle-1.2.2-py2.py3-none-any.whl
Collecting werkzeug>=0.11.15
  Using cached https://files.pythonhosted.org/packages/ce/42/3aeda98f96e85fd26180534d36570e4d18108d62ae36f87694b476b83d6f/Werkzeug-0.16.0-py2.py3-none-any.whl
Collecting setuptools>=41.0.0
  Using cached https://files.pythonhosted.org/packages/ea/98/9db5ebb9c33716ab8c45ee355a300facb44575b9b6d68a95fa49e81a2f04/setuptools-41.5.0-py2.py3-none-any.whl
Collecting markdown>=2.6.8
  Using cached https://files.pythonhosted.org/packages/c0/4e/fd492e91abdc2d2fcb70ef453064d980688762079397f779758e055f6575/Markdown-3.1.1-py2.py3-none-any.whl
Collecting h5py
  Using cached https://files.pythonhosted.org/packages/0b/fa/bee65d2dbdbd3611702aafd128139c53c90a1285f169ba5467aab252e27a/h5py-2.10.0-cp36-cp36m-win_amd64.whl
Installing collected packages: wheel, numpy, six, google-pasta, keras-preprocessing, setuptools, protobuf, werkzeug, markdown, absl-py, grpcio, tensorboard, h5py, keras-applications, termcolor, gast, astor, tensorflow-estimator, wrapt, tensorflow, decorator, cloudpickle, tensorflow-probability
Successfully installed absl-py-0.8.1 astor-0.8.0 cloudpickle-1.2.2 decorator-4.4.1 gast-0.3.2 google-pasta-0.1.7 grpcio-1.24.3 h5py-2.10.0 keras-applications-1.0.8 keras-preprocessing-1.1.0 markdown-3.1.1 numpy-1.17.3 protobuf-3.10.0 setuptools-41.5.0 six-1.12.0 tensorboard-1.14.0 tensorflow-1.14.0 tensorflow-estimator-1.14.0 tensorflow-probability-0.7.0 termcolor-1.1.0 werkzeug-0.16.0 wheel-0.33.6 wrapt-1.11.2

Installation complete.
#10

OK, cool. The good news is it’s installing the correct versions in the correct place (the conda r-reticulate environment)!

The problem must therefore be that it’s loading the wrong python. Assuming the reticulate::py_config() and tensorflow::tf_config() outputs are the same as before, reticulate is using the main version of anaconda python, rather than the version of python in the r-reticulate environment.

Try this:

  • restart the R session and clear all (including hidden) objects
  • before you run any other line of code, do: reticulate::use_condaenv("r-reticulate", required = TRUE)
  • then try loading greta

If that doesn’t work, I’d try this:

  • restart the R session and clear all (including hidden) objects
  • before you run any other line of code, do: reticulate::use_python("C:/Users/Hertzog/Documents/.conda/envs/r-reticulate", required = TRUE)
  • then try loading greta

If neither of those work, could you please send the console output and results of reticulate::py_config() and tensorflow::tf_config() for both of those?

How reticulate picks which python to load by default eludes me somewhat, and it seems to be less likely to do the right thing for tensorflow when running on windows than on linux/OSX (though I may just have a biased sample)

#11

Ah! The first option did the trick! For the record here are the steps that I followed:

  1. Remove existing environments of tensorflow and reticulate:

  1. Re-starting R and clearing all files including hidden ones rm(list=ls(all.names = TRUE)) or using the broom icon on RStudio

  2. Running

  1. Re-starting R and clearing all files again

  2. Setting reticulate to use the right python version:

  1. Loading greta, library(greta)

Thanks Nick for your patience, now the next CRAN version of greta with TF2.0 can come out :slight_smile:

#12

Great!

Yes, hopefully will get the TF2 version out soon :slight_smile:

#13

Hi, I followed @lionel68 's steps but did not work.

However, after some trial and error I found that after reinstalling reticulate (dev version on github using devtools::install_github("rstudio/reticulate")), library(greta) will run and automatically install the latest tensorflow for me. After that, I just need to manually install the older tensorflow:

After that, restart R session and library(greta) loads normally for me :slight_smile: Hope this helps some of us out there.