pathlib destpath.exists() true even file does not exist

[NaN]

I have a Windows file server (Win Server 2003 Standard x64 Edition) that contains csv files.

On a Debian machine (9.13 Stretch) I mounted a Windows share pointing to abovementioned machine by adding this line in file /etc/fstab:
//192.168.254.10/DATA /mnt/fsdata cifs uid=postgres,username=<*user*>,password=<*pwd*>,iocharset=utf8,sec=ntlm 0 0

I am running a python script on the Debian machine to check if the csv files exist on the windows machine.

On the windows machine, when I “cut” or “drag” all the files away from folder, and run Path.exists() on /mnt/fsdata/IT/Servers/PostGres/CDR/, the result is always true. (=incorrect)

On the windows machine, when I delete all the files, and run Path.exists() on /mnt/fsdata/IT/Servers/PostGres/CDR/, the result is false. (=correct)

It seems that cut or drag is not recognized on Debian side, and python still “sees” the files as being there.

This is the code excerpt:

#!/usr/bin/env python3 import pathlib src = "/mnt/fsdata/IT/Servers/PostGres/CDR/" dest = src + "procd/" srcpath = pathlib.Path(src) for file in list(srcpath.glob('*.csv')): destpath = pathlib.Path(dest + file.name) # check if file already exists in /procd folder: if destpath.exists(): # something happens...

What can be done to avoid this?

I searched a lot online but found no answers.

Thank you
NaN

[Larz60+]

pathlib will return the 'address' of where you want a file to be.
To see if file is there, use exists.

example (untested):

from pathlib import Path homepath = Path('.') datapath = homepath / 'data' datapath.mkdir(exist_ok=True) # make data directory (only if it is not already there) myfile = datapath / 'myfile.text' # check if file exists: if myfile.exists(): with myfile.open() as fp: data = myfile.read() else: print("myfile.txt does not exist")

[Gribouillis]

I suspect it has something to do with the way samba servers work, or samba clients. Some answers on the web, such as this one indicate that it could be a cache problem. I'm not proficient in these samba issues, but you can perhaps either fine tune the configuration of the shared directory on the Windows side or find a way for the Linux client to force the server to reread the contents of the directory. This link could be helpful too.

[NaN]

I suspect it has something to do with the way samba servers work, or samba clients. Some answers on the web, such as this one indicate that it could be a cache problem. I'm not proficient in these samba issues, but you can perhaps either fine tune the configuration of the shared directory on the Windows side or find a way for the Linux client to force the server to reread the contents of the directory. This link could be helpful too.

I added cache=none in /etc/fstab and restarted the machine. Nothing changed.
My opinion is that the problem is on python side, because when I ls the folder it is empty. After that, running the py script still gives destpath.exists() as being true.

I added following lines to try to open files:

with destpath.open() as f: print("File name: " + str(destpath))

it returns an error:

Error:
An error occurred: [Errno 2] No such file or directory: '/mnt/fsdata/IT/Servers/PostGres/CDR/procd/Trunks-2020-11-01.csv'

Could it be a bug in python? Who can I contact for that?

[Gribouillis]

You could perhaps first perform a os.stat() call on the file and print all the fields of the returned stat_result object to see what it contains. A bug in Python is by far the least plausible explanation.

[NaN]

stats are as follows:

Output:
os.stat_result(st_mode=33261, st_ino=1407374883714381, st_dev=41, st_nlink=1, st_uid=118, st_gid=0, st_size=1621, st_atime=1606730312, st_mtime=1606730312, st_ctime=1606730608) os.stat_result(st_mode=33261, st_ino=1125899907003726, st_dev=41, st_nlink=1, st_uid=118, st_gid=0, st_size=6249, st_atime=1606730312, st_mtime=1606730312, st_ctime=1606730608) os.stat_result(st_mode=33261, st_ino=1125899907003737, st_dev=41, st_nlink=1, st_uid=118, st_gid=0, st_size=8594, st_atime=1606730312, st_mtime=1606730312, st_ctime=1606730608)

etc...

[DeaD_EyE]

This kind of checks leads very often into problems. More than you think.

# check if file exists: if myfile.exists(): # <--- file could exist during this moment # <--- maybe the file is now deleted, has changed permission or something else. with myfile.open() as fp: # <-- will definitely raise an Exception if there is a problem data = myfile.read() # <-- could also raise an Exception

Don't ask for permission, ask for forgiveness:

from pathlib import Path ... myfile = Path("C:") # try other not working Paths ... try: with myfile.open() as fp: data = fp.read() except FileNotFoundError: print("File not found") except PermissionError: print("I do not have the permission to access", myfile) except UnicodeDecodeError: print("Could not decode UTF-8. Binary file or wrong encoding?")

Output:
I do not have the permission to access C:

Try this with binary files and you'll get a UnicodeDecodeError.
Try this with your Samba-Share. It should raise FileNotFoundError, but maybe it's an OSError.
Just try it and observe which Exception you get.

[Gribouillis]

What happens if you try os.listdir(...) or list(os.scandir(...)) and if you mix calls to subprocess.check_output(['ls', ...]) or subprocess.check_output("ls ...", shell=True) in between?

[NaN]

I tried your suggestion:

from pathlib import Path src = "/mnt/fsdata/IT/Servers/PostGres/CDR/" dest = src + "procd" # define the path currentDirectory = Path(src) currentPattern = "*.csv" destDirectory = Path(dest) for currentFile in currentDirectory.glob(currentPattern): destFile = destDirectory / currentFile.name if destFile.exists(): try: with destFile.open() as fp: data = fp.read() print(data) except FileNotFoundError: print("File not found") except PermissionError: print("I do not have the permission to access", myfile) except UnicodeDecodeError: print("Could not decode UTF-8. Binary file or wrong encoding?")

I get File not found.
I am the only user manipulating files in these folders. It is a test environment.
What am I doing wrong?

[NaN]

DeaD_EyE

OK, I tried your "Don't ask for permission, ask for forgiveness"-approach and rewrote my script:

#!/usr/bin/env python3.8 import pathlib from pathlib import Path src = "/mnt/fsdata/IT/Servers/PostGres/CDR/" dest = src + "procd" dup = src + "dup" # define the paths: currentDirectory = pathlib.Path(src) destDirectory = pathlib.Path(dest) dupDirectory = pathlib.Path(dup) currentPattern = "*.csv" for currentFile in currentDirectory.glob(currentPattern): destFile = destDirectory / currentFile.name try: with destFile.open() as fp: data = fp.read() except IOError: currentFile.rename(destFile) print('File moved to procd.') continue # Move file to dup folder: duppath = dupDirectory / currentFile.name currentFile.rename(duppath) print('File moved to dup.') print("Finished !")

When I upload files to /mnt/fsdata/IT/Servers/PostGres/CDR everything works fine.
Files are moved to /procd.
Now, when I drag those files from /procd back to /mnt/fsdata/IT/Servers/PostGres/CDR, nothing happens. The files remain where they are. It is as if the drag and drop never happened! (note that they are not moved to /dup either...)
What I basicaly want to achieve here is that I import files to /CDR folder. Check if they are not already in /procd. (if already in /procd, move them to /dup) Import them into a database. Move them to /procd.
What's wrong?