Summary:

Since Mathematica version 6, the PacletManager seems to have engineered an escalating hijack of the $Path variable. Add-on packages need to have a PacletInfo.m file for their documentation to function correctly. However, use of this file also seems to trigger an override of the $Path variable's control of file loading. This breaks some of the functionality of Wolfram Workbench when using Mathematica 9 and 10.

Examples:

Suppose, as a minimal example, I have a Test directory in my $UserBaseDirectory/Applications directory that contains a PacletInfo.m file that reads

Paclet[ Name -> "Test", Version -> "0.0.1" ] 

as well as a file foo.txt containing the text "Applications directory".

Also, in my $HomeDirectory I have another Test directory containing just a text file foo.txt reading "Home directory".

Now, Mathematica 6 behaves as I would expect -- if I empty the $Path variable, Get can't find either foo.txt file, and if I put the $HomeDirectory in the $Path it will find the file in the $HomeDirectory (note: clearing the $Path variable will interfere with stuff like searching the documentation, so you probably want to run these in a fresh kernel and then restart):

$Path = {}; Get["Test/foo.txt"] (* During evaluation of In[2]:= Get::noopen: Cannot open Test/foo.txt. *) (* $Failed *) $Path = {$HomeDirectory}; Get["Test/foo.txt"] (* "Home directory" *) 

In Mathematica 7 or 8, there is a different behavior -- even with an empty $Path it will find foo.txt in the Applications directory, but with $HomeDirectory in the path it will find foo.txt in the $HomeDirectory, as before:

$Path = {}; Get["Test/foo.txt"] (* "Applications directory" *) $Path = {$HomeDirectory}; Get["Test/foo.txt"] (* "Home directory" *) 

In Mathematica 9 or 10, however, it always finds the text file in the Applications directory, even if only the $HomeDirectory is in the $Path:

$Path = {}; Get["Test/foo.txt"] (* "Applications directory" *) $Path = {$HomeDirectory}; Get["Test/foo.txt"] (* "Applications directory" *) 

With no PacletInfo.m file we get the expected (version 6) behavior in all versions. As a side note, there is no effect on the results if we change the name of the Test directory in the $UserBaseDirectory/Applications directory to something else; Get behaves as if foo.txt were in a directory with whatever name is listed in the PacletInfo.m file.

Discussion:

What seems to be happening is that, when a new kernel is started (or if we manually call PacletManager`RebuildPacletData[]), a list of paths is scanned for PacletInfo.m files, and the Name entries in those files are registered as directory search paths, corresponding to the actual directories in which the PacletInfo.m files are located. In recent versions these paths take precedence over the search paths in $Path.

The behavior in versions 9 and 10 interferes with the way Wolfram Workbench is supposed to work. When launching Mathematica from the Workbench, it puts the local development folder at the beginning of the $Path variable directory list, so that the development version of the package is loaded, rather than any installed version in the Applications directory. When the PacletInfo.m file present, however, the installed version is always loaded.

My questions:

1. I'm curious if anyone has any insight as to whether the current behavior is considered desirable. Clearly a list of documentation directories has to be maintained, but is it necessary for the $Path to be overridden?

2. Does anyone have a workaround to get Workbench to load the development version of a package? The obvious method is to either overwrite or delete the installed version in the Applications directory, but this is clumsy. More elegant would be to manually add the development directory to the beginning of the paclet search path, as well as to the $Path, but I haven't found a way of doing that yet.