Backupy: Backup files from local machine to NAS and cloud using backupy and python

The concept is:

Backup is important. There are many backup tools in the world. There are also a backup manager tool written in python. This page presents an example of runnning backupy.
Here, LOCDIR is a local directory, in which we work, update, and add new files. NASDIR is a backup storage placed in the network. CLDDIR is a cloud service such as Box sync, One Drive, Google Backup and Sync, and Google Drive File Stream.
See also:

Python Matplotlib Tips: TarManager: combine many pickle files into some tar files using Python

TarManager: combine many pickle files into some tar files using Python

pythonmatplotlibtips.blogspot.com

Python Matplotlib Tips: Speed up generating figures by running external python script parallelly using Python and matplotlib.pyplot

This page shows my suggestion to process data and generate figure parallelly by running some external python script.

pythonmatplotlibtips.blogspot.com

Python Matplotlib Tips: Extract data from tar.gz and expand on RAM using tarfile library, then plot using matplotlib

This page shows an example of how to extract data from tar.gz file and expand not on HDD/SSD but on RAM, then plot some data using matplotlib. In some cases such as simulation, data logging,and image processing, you may have to deals with great many files composed of small files. In such situation, as you know, tar.gz file is efficient way to increase data transfer speed and decrease the number of the files. You have to extract some data from tar.gz file in order to generate figures using data. You don't have to deals with annoying intermediate files if you expand data not on HDD/SSD but directly on RAM as shown in this page.

pythonmatplotlibtips.blogspot.com

In [1]:

import backupy
import numpy as np
import random
import pickle
import shutil
import os
import sys
import platform
print('python: '+platform.python_version())

python: 3.7.6

Again, I want to use LOCDIR for working, NASDIR for backup, and CLDDIR for backup of NAS.

In [2]:

LOCDIR = './locdir'
NASDIR = './nasdir'
CLDDIR = './clddir'

Define dummy class to store variables and arrays.

In [3]:

class Dummy():
    def __init__(self, n, b, x, y):
        self.n = n
        self.b = b
        self.x = x
        self.y = y

Define a function to generate dummy data to subdir.

In [4]:

def savedummyfiles(Nfiles=5, savedir=LOCDIR, subdir=''):
    path = os.path.join(savedir, subdir)
    os.makedirs(path, exist_ok=True)
    smin, smax = np.infty, 0
    for idx in range(Nfiles):
        n = int(10**(0.5 + random.random() * 2.5))
        b = random.random()
        x = np.linspace(0, 1, n)
        y = x**b
        d = Dummy(n, b, x, y)
        with open(os.path.join(path, 'dat%04d.pkl' % idx), 'wb') as f:
            pickle.dump(d, f)
        s = os.path.getsize(os.path.join(path, 'dat%04d.pkl' % idx))
        smin, smax = min(smin, s), max(smax, s)
    print('%d files saved to %s' % (Nfiles, path))
    print('minimum file size: %d B' % smin)
    print('maximum file size: %d kB' % (smax / 1024))

In [5]:

savedummyfiles(subdir='sub1')
savedummyfiles(subdir='sub2')

5 files saved to ./locdir/sub1
minimum file size: 374 B
maximum file size: 6 kB
5 files saved to ./locdir/sub2
minimum file size: 390 B
maximum file size: 1 kB

Dummy files were saved. Let's check the directory.

In [6]:

%%sh
ls -R locdir

sub1
sub2

locdir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

The backupy can be used as command line script and also in the python script. Here, we use backupy in python script. For it, configuration dictionary must be given to BackupManager class.

In [7]:

cfg_loc2nas = {
    'source': LOCDIR,
    'dest': NASDIR,
    'mode': 'MIRROR',
    'noprompt': True,
    'nolog': True,
    'noarchive': True,
    'verbose': False,
    }
cfg_nas2cld = {
    'source': NASDIR,
    'dest': CLDDIR,
    'mode': 'MIRROR',
    'noprompt': True,
    'nolog': True,
    'noarchive': True,
    'verbose': False,
    }

First backup from LOCDIR to NASDIR

In [8]:

backup_man = backupy.BackupManager(cfg_loc2nas)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 10 (14.98 kB)
File: sub1/dat0000.pkl
                     Size: 1.08 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0001.pkl
                     Size: 1.24 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0002.pkl
                     Size: 1.21 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0003.pkl
                     Size: 374 B          Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0004.pkl
                     Size: 7.05 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0000.pkl
                     Size: 1.39 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0001.pkl
                     Size: 630 B          Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0002.pkl
                     Size: 390 B          Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0003.pkl
                     Size: 1.13 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0004.pkl
                     Size: 470 B          Modified: Sat Sep 19 20:58:46 2020
Destination Only (will be deleted): 0 (0 B)
Changed Files (delete dest and copy source -> dest): 0 (0 B)
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
File operations completed!                                                                                                                                                                                
Completed!

Out[8]:

0

As all files are new, all files were copied to NASDIR. Then, make backup from NASDIR to CLDDIR to improve resirience for disastor.

In [9]:

backup_man = backupy.BackupManager(cfg_nas2cld)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 10 (14.98 kB)
File: sub1/dat0000.pkl
                     Size: 1.08 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0001.pkl
                     Size: 1.24 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0002.pkl
                     Size: 1.21 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0003.pkl
                     Size: 374 B          Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0004.pkl
                     Size: 7.05 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0000.pkl
                     Size: 1.39 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0001.pkl
                     Size: 630 B          Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0002.pkl
                     Size: 390 B          Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0003.pkl
                     Size: 1.13 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub2/dat0004.pkl
                     Size: 470 B          Modified: Sat Sep 19 20:58:46 2020
Destination Only (will be deleted): 0 (0 B)
Changed Files (delete dest and copy source -> dest): 0 (0 B)
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
File operations completed!                                                                                                                                                                                
Completed!

Out[9]:

0

As all files are new, all files were copied to CLDDIR. Check the result.

In [10]:

%%sh
ls -R locdir nasdir clddir

clddir:
sub1
sub2

clddir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

clddir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir:
sub1
sub2

locdir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

nasdir:
sub1
sub2

nasdir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

nasdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

If some of the local files is overwritten, and some files were added, only the modified files and new files will be copied.

In [11]:

savedummyfiles(Nfiles=10, subdir='sub1') # 0-4: overwrite, 5-9: new file

10 files saved to ./locdir/sub1
minimum file size: 310 B
maximum file size: 3 kB

In [12]:

backup_man = backupy.BackupManager(cfg_loc2nas)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 5 (3.57 kB)
File: sub1/dat0005.pkl
                     Size: 358 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0006.pkl
                     Size: 1.85 kB        Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0007.pkl
                     Size: 310 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0008.pkl
                     Size: 374 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0009.pkl
                     Size: 678 B          Modified: Sat Sep 19 20:59:00 2020
Destination Only (will be deleted): 0 (0 B)
Changed Files (delete dest and copy source -> dest): 5 (6.44 kB)
File: sub1/dat0000.pkl
         Source      Size: 454 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 1.08 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0001.pkl
         Source      Size: 342 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 1.24 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0002.pkl
         Source      Size: 662 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 1.21 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0003.pkl
         Source      Size: 758 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 374 B          Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0004.pkl
         Source      Size: 3.08 kB        Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 7.05 kB        Modified: Sat Sep 19 20:58:46 2020
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
File operations completed!                                                                                                                                                                                
File operations completed!                                                                                                                                                                                
Completed!

Out[12]:

0

In [13]:

backup_man = backupy.BackupManager(cfg_nas2cld)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 5 (3.57 kB)
File: sub1/dat0005.pkl
                     Size: 358 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0006.pkl
                     Size: 1.85 kB        Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0007.pkl
                     Size: 310 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0008.pkl
                     Size: 374 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0009.pkl
                     Size: 678 B          Modified: Sat Sep 19 20:59:00 2020
Destination Only (will be deleted): 0 (0 B)
Changed Files (delete dest and copy source -> dest): 5 (6.44 kB)
File: sub1/dat0000.pkl
         Source      Size: 454 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 1.08 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0001.pkl
         Source      Size: 342 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 1.24 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0002.pkl
         Source      Size: 662 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 1.21 kB        Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0003.pkl
         Source      Size: 758 B          Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 374 B          Modified: Sat Sep 19 20:58:46 2020
File: sub1/dat0004.pkl
         Source      Size: 3.08 kB        Modified: Sat Sep 19 20:59:00 2020
           Dest      Size: 7.05 kB        Modified: Sat Sep 19 20:58:46 2020
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
File operations completed!                                                                                                                                                                                
File operations completed!                                                                                                                                                                                
Completed!

Out[13]:

0

The results follows:

In [14]:

%%sh
ls -R locdir nasdir clddir

clddir:
sub1
sub2

clddir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl
dat0005.pkl
dat0006.pkl
dat0007.pkl
dat0008.pkl
dat0009.pkl

clddir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir:
sub1
sub2

locdir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl
dat0005.pkl
dat0006.pkl
dat0007.pkl
dat0008.pkl
dat0009.pkl

locdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

nasdir:
sub1
sub2

nasdir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl
dat0005.pkl
dat0006.pkl
dat0007.pkl
dat0008.pkl
dat0009.pkl

nasdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

If some local files were deleted, the result is MIRRORD to the NASDIR and CLDDIR as the default mode setting is MIRROR.

In [15]:

for i in range(0, 7):
    os.remove(os.path.join(LOCDIR, 'sub1', 'dat%04d.pkl' % i))

In [16]:

backup_man = backupy.BackupManager(cfg_loc2nas)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 0 (0 B)
Destination Only (will be deleted): 7 (7.51 kB)
File: sub1/dat0000.pkl
                     Size: 454 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0001.pkl
                     Size: 342 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0002.pkl
                     Size: 662 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0003.pkl
                     Size: 758 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0004.pkl
                     Size: 3.08 kB        Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0005.pkl
                     Size: 358 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0006.pkl
                     Size: 1.85 kB        Modified: Sat Sep 19 20:59:00 2020
Changed Files (delete dest and copy source -> dest): 0 (0 B)
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
Completed!

Out[16]:

0

In [17]:

backup_man = backupy.BackupManager(cfg_nas2cld)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 0 (0 B)
Destination Only (will be deleted): 7 (7.51 kB)
File: sub1/dat0000.pkl
                     Size: 454 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0001.pkl
                     Size: 342 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0002.pkl
                     Size: 662 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0003.pkl
                     Size: 758 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0004.pkl
                     Size: 3.08 kB        Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0005.pkl
                     Size: 358 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0006.pkl
                     Size: 1.85 kB        Modified: Sat Sep 19 20:59:00 2020
Changed Files (delete dest and copy source -> dest): 0 (0 B)
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
Completed!

Out[17]:

0

In [18]:

%%sh
ls -R locdir nasdir clddir

clddir:
sub1
sub2

clddir/sub1:
dat0007.pkl
dat0008.pkl
dat0009.pkl

clddir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir:
sub1
sub2

locdir/sub1:
dat0007.pkl
dat0008.pkl
dat0009.pkl

locdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

nasdir:
sub1
sub2

nasdir/sub1:
dat0007.pkl
dat0008.pkl
dat0009.pkl

nasdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

If some of the local directory were deleted, the result is also MIRRORED to the NASDIR and CLDDIR.

In [19]:

shutil.rmtree(os.path.join(LOCDIR, 'sub1'))

In [20]:

%%sh
ls -R locdir

sub2

locdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

In [21]:

backup_man = backupy.BackupManager(cfg_loc2nas)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 0 (0 B)
Destination Only (will be deleted): 3 (1.36 kB)
File: sub1/dat0007.pkl
                     Size: 310 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0008.pkl
                     Size: 374 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0009.pkl
                     Size: 678 B          Modified: Sat Sep 19 20:59:00 2020
Changed Files (delete dest and copy source -> dest): 0 (0 B)
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
Completed!

Out[21]:

0

In [22]:

backup_man = backupy.BackupManager(cfg_nas2cld)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 0 (0 B)
Destination Only (will be deleted): 3 (1.36 kB)
File: sub1/dat0007.pkl
                     Size: 310 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0008.pkl
                     Size: 374 B          Modified: Sat Sep 19 20:59:00 2020
File: sub1/dat0009.pkl
                     Size: 678 B          Modified: Sat Sep 19 20:59:00 2020
Changed Files (delete dest and copy source -> dest): 0 (0 B)
Moved Files (will move files on dest to match source): 0 (0 B)
Starting mirror
Completed!

Out[22]:

0

In [23]:

%%sh
ls -R locdir nasdir clddir

clddir:
sub2

clddir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir:
sub2

locdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

nasdir:
sub2

nasdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

This is super dangerous for me. I usually working in the LOCDIR, and then the sub-directory (such as proj_A_2020aug, task_C_20200903) on my LOCDIR is deleted after the job finished to reduce storage usase of my local SSD. But I want to remain the sub-directory on NASDIR and CLDDIR to ensure data. The preferred usage for me is assign LOCDIR/sub-dir, NASDIR/sub-dir, and CLDDIR/sub-dir for source and dest as follows:

In [24]:

shutil.rmtree(os.path.join(LOCDIR, 'sub2'))

In current verson (v1.8.2, 2020 Jul 1st) of backupy, absent source will abort the backup with a message of Invalid source directory. Therefore, the files in NASDIR and CLDDIR is not deleted.

In [25]:

cfg_loc2nas.update({
    'source': os.path.join(LOCDIR, 'sub2'),
    'dest': os.path.join(NASDIR, 'sub2'),
})
backup_man = backupy.BackupManager(cfg_loc2nas)
backup_man.run()

Invalid source directory: ./locdir/sub2

An exception has occurred, use %tb to see the full traceback.

SystemExit

MYPATH/envs/py376/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3339: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

In [26]:

cfg_nas2cld.update({
    'source': os.path.join(NASDIR, 'sub2'),
    'dest': os.path.join(CLDDIR, 'sub2'),
})
backup_man = backupy.BackupManager(cfg_nas2cld)
backup_man.run()

Scanning completed!                                                                                                                                                                                       
Scanning completed!                                                                                                                                                                                       
Source Only (will be copied to dest): 0 (0 B)
Destination Only (will be deleted): 0 (0 B)
Changed Files (delete dest and copy source -> dest): 0 (0 B)
Moved Files (will move files on dest to match source): 0 (0 B)
Directories already match, completed!

Out[26]:

0

In [27]:

%%sh
ls -R locdir nasdir clddir

clddir:
sub2

clddir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir:

nasdir:
sub2

nasdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

The backupy can be run as the command line script. This is preferred than running in the python because output line number is enormous in some cases. Let's see the example.

In [28]:

savedummyfiles(subdir='sub1')
savedummyfiles(subdir='sub2')

5 files saved to ./locdir/sub1
minimum file size: 406 B
maximum file size: 11 kB
5 files saved to ./locdir/sub2
minimum file size: 358 B
maximum file size: 11 kB

In [29]:

%%writefile backupytest.py
import backupy
import subprocess

if __name__ == '__main__' :
    opttxt = '--noprompt'
    locdst = (
        ('./locdir/sub1',
         './nasdir/sub1',),
        ('./nasdir/sub1',
         './clddir/sub1',),
        ('./locdir/sub2',
         './nasdir/sub2',),
        ('./nasdir/sub2',
         './clddir/sub2'),
    )
    for idx, (loc, dst) in enumerate(locdst):
        cmd = 'backupy %s %s %s 1> stdout%04d.log 2> stderr%04d.log' \
                % (loc, dst, opttxt, idx, idx)
        proc = subprocess.check_call(cmd, shell=True)

Overwriting backupytest.py

In [30]:

%%capture capt
%run backupytest.py

In [31]:

%%sh
ls -R locdir nasdir clddir

clddir:
sub1
sub2

clddir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

clddir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir:
sub1
sub2

locdir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

locdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

nasdir:
sub1
sub2

nasdir/sub1:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl

nasdir/sub2:
dat0000.pkl
dat0001.pkl
dat0002.pkl
dat0003.pkl
dat0004.pkl