(following up from [here](https://codereview.stackexchange.com/questions/245670/scan-a-directory-for-files-and-load-it-in-memory-efficiently)
I am working on a little project where I need to scan all the config files present in a folder on the disk and load it in memory. Below are the steps:
- On the disk there is already a default `Records` folder which has all the default config files present. This is to fallback if `loadDefaultFlag` is enabled.
- There are also new config files present as a tar.gz file (max 100 MB size) in a remote url location which I need to download and store it on disk in secondary location only if `loadDefaultFlag` is disabled.
Depending on whether `loadDefaultFlag` is enabled or not - we will either load default local files already present on the disk or load it from secondary location (after downloading it from the remote url location). During server startup call goes to my `RecordManager` constructor where it checks whether `loadDefaultFlag` is enabled or not and basis on that it loads the file either from `Records` folder as mentioned in point 1 or download new configs from url as mentioned in point 2 and then load it in memory.
I have a `configKey` variable stored in `IConfiguration` object which has below data and that's what my below code uses to do above things.
{"loadDefaultFlag":"false", "remoteFileName":"abc-123.tgz", "reload":"false"}
Below is my code:
public class RecordManager
{
private const string _remoteUrl = "remote-url-from-where-to-download-new-configs";
private static string _remoteFileName;
private const string SecondaryLocation = "SecondaryConfigs";
private readonly IConfiguration _configuration;
private readonly string _localPath;
private IEnumerable<RecordHolder> _records;
public enum ConfigLocation { System, Local, Remote }
public RecordManager(IConfiguration configuration, string localPath)
{
if(configuration == null) { throw new ArgumentNullException(nameof(configuration)); }
if(localPath?.Length == 0) { throw new ArgumentNullException(nameof(localPath)); }
_localPath = localPath;
_configuration = configuration;
ChangeToken.OnChange(configuration.GetReloadToken, _ => ConfigChanged(), new object());
}
public RecordManager(IConfiguration configuration) : this(configuration, "Records") { }
public RecordManager LoadConfigurationsFrom(ConfigLocation location)
{
switch(location)
{
case ConfigLocation.Remote:
_records = GetConfigFromServer();
break;
case ConfigLocation.Local:
_records = GetConfigFromLocalFiles();
break;
case ConfigLocation.System:
_records = IsConfigFromServer() ? GetConfigFromServer() : GetConfigFromLocalFiles();
break;
}
return this;
}
public void Save()
{
// now load `_records` configs in memory here
// only called once you are ready to switch
}
private bool IsConfigFromServer()
{
string configValue = configuration["configKey"];
if (string.IsNullOrWhiteSpace(configValue)){ return false; }
var dcc = JsonConvert.DeserializeObject<RecordPojo>(configValue);
if(!bool.TryParse(dcc.loadDefaultFlag?.ToString(), out bool loadDefaultFlag)) { return false; }
_remoteFileName = dcc.remoteFileName;
return !loadDefaultFlag && !string.IsNullOrWhiteSpace(dcc.remoteFileName);
}
// download tar.gz file from remote server, store it on disk in secondary location
// uncompress tar.gz file, read it and return RecordHolder list back.
private IEnumerable<RecordHolder> GetConfigFromServer()
{
var isDownloaded = _fileHelper.Download($"{_remoteUrl}{_remoteFileName}", _secondaryLocation);
if(!isDownloaded) { yield return default; }
var isExtracted = _fileHelper.ExtractTarGz(_remoteFileName, _directory);
if(!isExtracted) { yield return default; }
foreach(var configPath in _fileHelper.GetFiles(directory))
{
if(!File.Exists(configPath)) { continue; }
var fileDate = File.GetLastWriteTimeUtc(configPath);
var fileContent = File.ReadAllText(configPath);
var pathPieces = configPath.Split(System.IO.Path.DirectorySeparatorChar, StringSplitOptions.RemoveEmptyEntries);
var fileName = pathPieces[pathPieces.Length - 1];
yield return new RecordHolder
{
Name = fileName,
Date = fileDate,
JDoc = fileContent
};
}
}
private IEnumerable<RecordHolder> GetConfigFromLocalFiles()
{
// read config files already present in default "Records" folder
// and return RecordHolder list back.
}
// this can be improved a lot to achieve below cases in proper way
private void ConfigChanged()
{
string configValue = _configuration["configKey"];
if (string.IsNullOrWhiteSpace(configValue)) { return; }
var dcc = JsonConvert.DeserializeObject<ConsulConfig>(configValue);
bool.TryParse(dcc.loadDefaultFlag?.ToString(), out bool loadDefaultFlag);
bool.TryParse(dcc.reloadConfig?.ToString(), out bool reloadConfig);
_remoteFileName = dcc.remoteFileName;
if (switchConfig) { Save(); }
if (loadDefaultFlag) { _records = GetConfigFromLocalFiles(); }
else { _records = GetConfigFromServer(); }
}
}
This is how I use it as a fluent api and during server startup this will be called as it is:
new RecordManager(configuration)
.LoadConfigurationsFrom(RecordManager.ConfigLocation.Remote)
.Save();
**Question:**
Now as you can see I have a `ChangeToken.OnChange` notification enabled in my constructor where I need to do something whenever my configuration (configKey) is changed and it will invoke my `ConfigChanged` method. Basically after server startup is done and configs is loaded up in memory with above code then someone can tell us to download new configs again and then load it in memory. For example:
Below are the cases that could have happen after first server startup:
Case 1 - Someone might change `configKey` to below value:
{"loadDefaultFlag":"false", "remoteFileName":"abc-124.tgz", "reload":"false"}
So now I need to download `abc-124.tgz` file from `remoteServer` and store it on disk but don't need to switch to this new config yet in memory (by calling save method).
Case 2 - After sometime, someone will change reload value to true:
{"loadDefaultFlag":"false", "remoteFileName":"abc-124.tgz", "reload":"true"}
And then I will check whether we have new configs `abc-124.tgz` already downloaded without any issues - if yes, then we will switch to use new configs by calling save method. And if this new config is already downloaded and switched then we will not do anything.
Case 3 - Maybe someone decides to use local files so they can enable `loadDefaultFlag` to true. And then we need to do exact same thing like case1 and then we can switch later with another step:
{"loadDefaultFlag":"true", "remoteFileName":"abc-124.tgz", "reload":"false"}
Opting for a code review here. I am specifically interested in the way I have designed and implemented my code. I am sure there must be a better way to rewrite the `ConfigChanged` method code in a better way that can handle all those above cases efficiently.