I'm now able to answer my own question, by piecing together various other stackoverflow answers and blog posts. In the end, the solution was to:
- create a virtual environment,
- create a development version of the working package within that environment, and
- create a jupyter/ipython kernel for the the environment to use in JupyterHub.
Specifically, this is how I achieved those things and the hiccups along the way:
Creating the Virtual Environment
This is all standard conda virtual environments. Install virtualenv. If conda is already installed, use
conda install virtualenv
pip install will not work in this case. Then create a directory 'virtualenvs' in your home directory to store the environment. The directory name and environment name don't matter.
virtualenv ~/virtualenvs/my_env
creates a virtual environment, my_env. Now, activate that with
source ~/virtualenvs/my_env/bin/activate
The prompt will show the new environment.
Create Development Package
Navigate to the package you want to develop in and run
python setup.py develop
In my case, this chugged for quite a long time and gave endless warnings about numpy. I didn't investigate this further because in the end it worked, and it seems others reported similar issues with develop and numpy.
When its done, you should get a message like
Using <your_home>/virtualsenvs/my_env/lib/python3.6/site-packages Finished Processing dependencies for way-cool-package==0.01
You can navigate to that location and confirm existence of an egg.lnk file, and you can also check the package is 'installed' with
pip freeze
At this point, with the python interpreter you can test that your module is accessible. I added a new print_hello method to my module and then checked that I could import that. The first time I did all this, I didn't have the order correct, and as a result, python was only importing the site-installed package, not my new enhancements.
We work in JupyterHub a lot, so this wasn't good enough for our team. If you try to use ipython on the command line here, it will nag. Next step..
Create Jupyter Kernel
Within the new virtual environment, check if ipykernel module is installed by using pip freeze. Otherwise,
pip install ipykernel
Create a new Jupyter kernel with the name new_kernel by
python -m ipykernel install --user --name=new_kernel
This will install the kernel in your home directory and report
Installed kernelspeck new_kernel in <your_home>/.local/share/jupyter/kernels/new_kernel
Now, refresh your JupyterHub Notebook, open an new one, and you should have 'new_kernel' as an option.
The Last Gotcha
The last gotcha I encountered was in a colleague's package on which I have a dependency. They used __file__ in their code to create their doc string. When trying to use my own modules in the new kernel, I hit a 'NotADirectoryError' which puzzled me. After all, their code was pip-installed site-wide and had no issues. Then, I noticed in the setup tools documentation that instead of using references like __file__, which make an assumption about the path, you should use pkg_resource.resouce_string(). I made that change to their code, and all was perfect.
Here are some of the resources that I used:
sys.path.append('absolute_path_to_new_package_dir/')?