Unable to resolve FileNotFoundError

[llarkin6]

Hello All,

I am very much so a novice to python and programming in general. I am attempting to use this data processing program that someone else wrote, for an experiment I'm doing. I'm running it in a juypter notebook, and was using it without issue until the other day.

As part of the program, it creates an output folder and puts all the files it creates there. It's able to create this output folder, but then throws the error. The program does contain code to throw errors if it's unable to access the input and output pathways, the input file. And this is a different error from those to my eyes.

I've tried re-writing the input/output file pathways in case that was a problem. The files are all stored on Box Drive, but I also tried running it with the files locally on my PC. I tried it with other files that worked previously, renaming the input file, comparing the input file to previous ones to make sure it was formatted in the same way. I was able to get it to work yesterday after closing out of everything and restarting my computer, creating a new juypter notebook. I don't recall all that I did, but I attempted to repeat it today to no avail.

Here is the program and error:

# %load dataCompile.py import os import sys import numpy as np from pathVisualization import PathVisualization from simulation import Sim class DataProcessor: def __init__(self): #Uncomment machine type based on data source #self.machineType = "sim" #self.machineType = "clinostat" #self.machineType = "rpm-integ" #self.machineType = "rpm-ard" self.machineType = "CSV" #Set range for average magnitude (seconds) self.minSeg = 2000 self.maxSeg = 3000 #Set data output path outParent = "C:\\Users\\Linaria\\Box\\Meyer lab documents\\Meyer lab Data\\Linaria\\Clinostat Project\\3D-Clinostat-main(1)\\3D-Clinostat-main\\Programs\\Output\\" #SET OUTPUT FILE PATH if self.machineType != "sim": self.inPath = "C:\\Users\\Linaria\\Box\\Meyer lab documents\\Meyer lab Data\\Linaria\\Clinostat Project\\3D-Clinostat-main(1)\\3D-Clinostat-main\\Programs\\Input\\" #SET INPUT FILE PATH self.inName = "ADX_5.csv" #SET FILE NAME try: testOpen = open(self.inPath + self.inName, "r") testOpen.close() except FileNotFoundError: print("\nERROR: Input file not found: " + self.inPath + self.inName + "\n") sys.exit() if self.machineType != "sim": self.outPath = outParent + self.inName[:len(self.inName)-4] + "\\" else: self.outPath = outParent + self.inName + "\\" print(self.outPath) else: #Applies only for Computer Model self.innerV = 1.5 #SET INNER VELOCITY self.outerV = 3.875 #SET OUTER VELOCITY self.inName = str(self.innerV) + "-" + str(self.outerV) self.outPath = outParent + self.inName + "\\" if os.path.isdir(self.outPath) == False: os.makedirs(self.outPath) def outputDataFile(self): """Outputs processed data file""" self.__getVectors() xTimeAvg, yTimeAvg, zTimeAvg = self.__getTimeAvg() magList = self.__getMagnitude(xTimeAvg, yTimeAvg, zTimeAvg) avgMagSeg = self.__getMagSeg(magList) disScore = self.__getDisScore() fileOut = open(self.outPath + 'processedAccelData.txt', 'w+') if self.machineType != 'sim': fileOut.write(self.inName) else: fileOut.write('Inner: ' + str(self.innerV) + '\t' + 'Outer: ' + str(self.outerV) + '\n') fileOut.write('Segment: ' + str(self.minSeg) + ":" + str(self.maxSeg) + '\t' + str(avgMagSeg) + '\t' + '\t\t\t' + 'Dispersion: ' + '\t' + str(disScore) +'\n') fileOut.write('Time' + '\t' + 'X Vector' + '\t' + 'Y Vector' + '\t' + 'Z Vector' + '\t' + 'X Time Avg' + '\t' + 'Y Time Avg' + '\t' + 'Z Time Avg' + '\t' + 'Magnitude' + '\n') for i in range(len(self.x)): fileOut.write(str(self.time[i]) + '\t' + str(self.x[i]) + '\t' + str(self.y[i]) + '\t' + str(self.z[i]) + '\t' + str(xTimeAvg[i]) + '\t' + str(yTimeAvg[i]) + '\t' + str(zTimeAvg[i]) + '\t' + str(magList[i]) + '\n') fileOut.close() print("Average Magnitude: " + str(avgMagSeg)) print("Distribution: " + str(disScore)) def __getVectors(self): """Directs where to get data from""" if self.machineType == 'sim': self.time, self.x, self.y, self.z = self.__getSimAccelData() elif self.machineType == 'clinostat' or self.machineType == 'rpm-ard': self.time, self.x, self.y, self.z = self.__getArdAccelData() elif self.machineType == 'CSV': self.time, self.x, self.y, self.z = self.__getCSVAccelData() else: self.time, self.x, self.y, self.z = self.__getRPMAccelData() def __getCSVAccelData(self): x, y, z, time = [], [], [], [] x = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(1)) y = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(2)) z = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(3)) time = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(0)) return time, x, y, z def __getArdAccelData(self): """Reads data from file formatted by the Arduino UNO Accelerometer""" dataIn = open(self.inPath + self.inName, 'r') x, y, z, time = [], [], [], [] for line in dataIn.readlines(): lineSplit = line.split('\t') if len(lineSplit) == 4: x.append(float(lineSplit[0])) y.append(float(lineSplit[1])) z.append(float(lineSplit[2])) unTime = float(lineSplit[3]) / 1000 time.append(unTime) dataIn.close() return time, x, y, z def __getRPMAccelData(self): """Reads data from file formatted by the RPM 2.0""" dataIn = open(self.inPath + self.inName, 'r') x, y, z, time = [], [], [] ,[] for line in dataIn.readlines(): lineFirstSplit = line.split('\t') splitVectors = lineFirstSplit[1] vectorList = splitVectors.split(' ') x.append(float(vectorList[0])) z.append(float(vectorList[1])) y.append(float(vectorList[2])) for i in range(len(x)): time.append(i) dataIn.close() return time, x, y, z def __getSimAccelData(self): """Gets data from computer model""" simInnerV = float(self.innerV) simOuterV = float(self.outerV) vectorSim = Sim() time, x, y, z = vectorSim.gVectorData(0, self.maxSeg, simInnerV, simOuterV) return time, x, y, z def __getTimeAvg(self): """Calculates Time Average""" xTimeAvg = [] xTempList = [] for xIter in self.x: xTempList.append(xIter) xTimeAvg.append(np.mean(xTempList)) yTimeAvg = [] yTempList = [] for yIter in self.y: yTempList.append(yIter) yTimeAvg.append(np.mean(yTempList)) zTimeAvg = [] zTempList = [] for zIter in self.z: zTempList.append(zIter) zTimeAvg.append(np.mean(zTempList)) return xTimeAvg, yTimeAvg, zTimeAvg def __getMagnitude(self, xTimeAvg, yTimeAvg, zTimeAvg): """Calculates magnitude of time-averaged acceleration vector""" magList = [] for i in range(len(self.x)): xIter = xTimeAvg[i] yIter = yTimeAvg[i] zIter = zTimeAvg[i] mag = (xIter ** 2 + yIter ** 2 + zIter ** 2) ** 0.5 magList.append(mag) return magList def __getMagSeg(self, magList): """Calculates avergae magnitude between segmentd""" magSegList = magList[self.minSeg:self.maxSeg] if len(magList) < self.minSeg: print("\nERROR: Segment begins after data ends - " + str(len(magList)) + " sec\n") sys.exit() elif len(magSegList) < (self.maxSeg - self.minSeg): print("\nWARNING: Not enough data for segment - " + str(len(magList)) + " sec\n") return np.mean(magList[self.minSeg:self.maxSeg]) def __getDisScore(self): xSeg = self.x[:self.maxSeg] ySeg = self.y[:self.maxSeg] zSeg = self.z[:self.maxSeg] path = PathVisualization(self.inName, xSeg, ySeg, zSeg) disScore = path.getDistribution() return disScore def copyRawData(self): if self.machineType == "sim": print("\nWARNING: Machine Type = Sim - No files were copied\n") return dataIn = open(self.inPath + self.inName, 'r') copyData = open(self.outPath + self.inName[:len(self.inName)-4] + '-Copy' + self.inName[len(self.inName)-4:], 'w+') copyData.write(self.inPath + self.inName + '\n\n') for line in dataIn.readlines(): copyData.write(line) copyData.close() dataIn.close() def createGraphs(self): self.__getVectors() xTimeAvg, yTimeAvg, zTimeAvg = self.__getTimeAvg() magnitude = self.__getMagnitude(xTimeAvg, yTimeAvg, zTimeAvg) if self.machineType != 'sim': graphPath = PathVisualization(self.inName, self.x, self.y, self.z, saveFile=self.outPath + self.inName[:len(self.inName)-4]) else: graphPath = PathVisualization(self.inName, self.x, self.y, self.z, saveFile=self.outPath + self.inName) graphPath.createPathShadowFig(mode='save', legend=False) graphPath.createVectorFig(self.time, mode='save') graphPath.createTimeAvgFig(xTimeAvg, yTimeAvg, zTimeAvg, self.time, mode='save') graphPath.createMagFig(magnitude, self.time, mode='save') def createSegGraphs(self): self.__getVectors() xSeg = self.x[2000:self.maxSeg] ySeg = self.y[2000:self.maxSeg] zSeg = self.z[2000:self.maxSeg] if self.machineType != 'sim': comparePaths = PathVisualization(self.inName, xSeg, ySeg, zSeg, saveFile=self.outPath + self.inName[:len(self.inName)-4] + '-Segment') else: comparePaths = PathVisualization(self.inName, xSeg, ySeg, zSeg, saveFile=self.outPath + self.inName + '-Segment') comparePaths.createPathFig(mode='save') if __name__ == "__main__": process = DataProcessor() process.outputDataFile() process.createGraphs() process.createSegGraphs() process.copyRawData()

Error:
--------------------------------------------------------------------------- FileNotFoundError Traceback (most recent call last) Cell In[4], line 239 237 if __name__ == "__main__": 238 process = DataProcessor() --> 239 process.outputDataFile() 240 process.createGraphs() 241 process.createSegGraphs() Cell In[4], line 50, in DataProcessor.outputDataFile(self) 48 def outputDataFile(self): 49 """Outputs processed data file""" ---> 50 self.__getVectors() 51 xTimeAvg, yTimeAvg, zTimeAvg = self.__getTimeAvg() 52 magList = self.__getMagnitude(xTimeAvg, yTimeAvg, zTimeAvg) Cell In[4], line 81, in DataProcessor.__getVectors(self) 79 self.time, self.x, self.y, self.z = self.__getArdAccelData() 80 elif self.machineType == 'CSV': ---> 81 self.time, self.x, self.y, self.z = self.__getCSVAccelData() 82 else: 83 self.time, self.x, self.y, self.z = self.__getRPMAccelData() Cell In[4], line 87, in DataProcessor.__getCSVAccelData(self) 85 def __getCSVAccelData(self): 86 x, y, z, time = [], [], [], [] ---> 87 x = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(1)) 88 y = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(2)) 89 z = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(3)) File ~\anaconda3\Lib\site-packages\numpy\lib\_npyio_impl.py:1397, in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin, encoding, max_rows, quotechar, like) 1394 if isinstance(delimiter, bytes): 1395 delimiter = delimiter.decode('latin1') -> 1397 arr = _read(fname, dtype=dtype, comment=comment, delimiter=delimiter, 1398 converters=converters, skiplines=skiprows, usecols=usecols, 1399 unpack=unpack, ndmin=ndmin, encoding=encoding, 1400 max_rows=max_rows, quote=quotechar) 1402 return arr File ~\anaconda3\Lib\site-packages\numpy\lib\_npyio_impl.py:1012, in _read(fname, delimiter, comment, quote, imaginary_unit, usecols, skiplines, max_rows, converters, ndmin, unpack, dtype, encoding) 1010 fname = os.fspath(fname) 1011 if isinstance(fname, str): -> 1012 fh = np.lib._datasource.open(fname, 'rt', encoding=encoding) 1013 if encoding is None: 1014 encoding = getattr(fh, 'encoding', 'latin1') File ~\anaconda3\Lib\site-packages\numpy\lib\_datasource.py:192, in open(path, mode, destpath, encoding, newline) 155 """ 156 Open `path` with `mode` and return the file object. 157 (...) 188 189 """ 191 ds = DataSource(destpath) --> 192 return ds.open(path, mode, encoding=encoding, newline=newline) File ~\anaconda3\Lib\site-packages\numpy\lib\_datasource.py:529, in DataSource.open(self, path, mode, encoding, newline) 526 return _file_openers[ext](found, mode=mode, 527 encoding=encoding, newline=newline) 528 else: --> 529 raise FileNotFoundError(f"{path} not found.") FileNotFoundError: ADX_5.csv not found.

Thank you all in advance for your help!

[noisefloor]

Hi,

based on the error message, at one point the inFile named "ADX_5.csv" is not found. As the attribute inFile is bonded just to the file name, the script excepts the in file, your CSV file, in the same directory as you script. Otherwise, you need to add the full path to the inFile attribute.

The script is a bit wild, not following Python's naming conventions and, specifically for your problem, uses concatenating strings by +, which is considered bad style and is prone to errors. For working with paths, use the pathlib module, which ships with Python and is the current standard. For formatting strings, use either f-Strings or the older .format method of strings.

When opening files for reading or writing, it is better to use the with statement wherever possible. By doing this, Python will automatically take care of closing the file properly without the need to call the file's close method yourself.

When opening text-based files for reading, it's better to explicitly state the encoding of the file so Python don't have to it. The latter may work, but no guarantee for that. At one point of the program, tab-separated data is written to a file. You may want to check here whether the csv module - which ships with Python - isn't the better choice over concatenating strings manually using +.

To make the script more flexible, it may be a good option being able to pass the required in and out file names as command line options. Or alternatively prompt for both.

Regards, noisefloor

[deanhystad]

The program checks if the file exists here:

 try: testOpen = open(self.inPath + self.inName, "r") testOpen.close() except FileNotFoundError:

This tests if a file named self.inName (ADX_5.csv) exists in a folder named:

C:\\Users\\Linaria\\Box\\Meyer lab documents\\Meyer lab Data\\Linaria\\Clinostat Project\\3D-Clinostat-main(1)\\3D-Clinostat-main\\Programs\\Input\\

The program does not report an error here, so there must be aan ADX_5 file in that folder.

The program doesn't use inPath in _getCSVAccelData when you load accel data from the csv file. It only uses self.inName.

 def __getCSVAccelData(self): x, y, z, time = [], [], [], [] x = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(1)) y = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(2)) z = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(3)) time = np.loadtxt(self.inName, delimiter=',', skiprows=1, dtype=np.float32, usecols=(0)) return time, x, y, z

This tries to open the ADX_5.csv file in the current working dictionary. The error message says the file isn't there.

Are you running the program from a different directory? When this worked you probably ran the file for the inpath folder ("C:\\Users\\Linaria\\Box\\Meyer lab documents\\Meyer lab Data\\Linaria\\Clinostat Project\\3D-Clinostat-main(1)\\3D-Clinostat-main\\Programs\\Input\\"). Maybe using a command like:

Output:
python ..\dataCompile.py

Or do you run the program by double clicking on the file icon in the file browser?

When the program doesn't work, you must be starting the program in a folder that doesn't have an ADX_5.csv file.
Did the ADX_5.csv file get deleted or are you starting the program in a different way?

[Pedroski55]

Let's practice finding a file!

from pathlib import Path # this file is actually in /home/peterr/temp/ file2 = 'datasets.txt' p2f = Path(file2) path2file = file2 # loop until you give an existing path # with any luck it may also be the file you want! while not p2f.exists(): print('To escape from this while loop, you must a) engage brain b) give a genuine path to the file you want.') print('You do know what file you want, correct?') print(f'cwd is {Path.cwd()}') print(f'No such file in {path2file}') path2file = input('Please enter the correct path to the file you want to open! ') p2f = Path(path2file) print(f'You set the path to {p2f}')

If you enter an existing path, you can escape the while loop! And you may have the file you want!

Output:
cwd is /home/peterr/PVE No such file in datasets.txt Please enter the correct path to the file you want to open! temp/datasets.txt You set the path to temp/datasets.txt cwd is /home/peterr/PVE No such file in temp/datasets.txt Please enter the correct path to the file you want to open! /peterr/temp/datasets.txt You set the path to /peterr/temp/datasets.txt cwd is /home/peterr/PVE No such file in /peterr/temp/datasets.txt Please enter the correct path to the file you want to open! /home/peterr/temp/datasets.txt You set the path to /home/peterr/temp/datasets.txt

[llarkin6]

lol thank you all for the info!

It is apparent to me that I have a lot to learn when it comes to this. I got this program to work, which is awesome!
However, the output is quite off from what I'd expect, and so I'm trying to confirm that it is processing the data in the way that is should.
And in my infinite ignorance, I don't fully follow all that's happening in it? I'm currently going through it step by step trying to understand what exactly it is doing and if that's following the intended logic.

I did confirm that it is putting the data from my files in the arrays as expected, but from there I'm kind of at a loss. I'm trying to follow the path of the functions being called and I'm stuck at when the __getTimeAvg function is called within the outputDataFile function, specifically this line in __getTimeAvg:

 def __getTimeAvg(self): """Calculates Time Average""" xTimeAvg = [] xTempList = [] for xIter in self.x: xTempList.append(xIter) xTimeAvg.append(np.mean(xTempList))

I see that xIter is declared in the __getMagnitude function, but that is called after __getTimeAvg, so how is __getTimeAvg using a variable that hasn't been declared yet? Obviously my understanding of the situation is incorrect or imcomplete , but I can't figure out how exactly . I tried adding a line to get an idea of what xIter contains to figure out its function like this:

 def outputDataFile(self): """Outputs processed data file""" self.__getVectors() xTimeAvg, yTimeAvg, zTimeAvg = self.__getTimeAvg() print(str(len(xIter) + "\n") magList = self.__getMagnitude(xTimeAvg, yTimeAvg, zTimeAvg) avgMagSeg = self.__getMagSeg(magList) disScore = self.__getDisScore()

And that did give me an error of xIter being not defined. (I first did the length of it in case it was extraordinarily long). So what exactly is happening here then? I apologize this is probably such a noob question, but I am forever grateful for people's help.

[deanhystad]

In this code:

 for xIter in self.x:

xIter is a local variable that is assigned to a value in self.x. First time through the loop xIter = self.x[0], second time through the look xIter = self.x[1] and so on for each value in self.x.

Since xIter is a local variable, it can only be seen inside __getTimeAvg(). The variable is created the first time the loop executes. Once the function is complete, the variable disappears.

[noisefloor]

@llarkin6: to get a base understanding of Python, I strongly suggest to work through the official Python tutorial, which can be found at docs.python.org . This should help to get at least a basic understanding of Python, functions, iteration in Python, scope of variables and other basics.

Regards, noisefloor