Python Forum
CSV file created is huge in size. How to reduce the size?
Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
CSV file created is huge in size. How to reduce the size?
#1
I have the below code used to create a CSV file:

import csv srcFile = open("C:\Users\XXXX\Expenses.csv","r") data = srcFile.read() srcList = data.split("\n") colNames = srcList[0].split(",") header = srcList[1] sheetData = srcList[2:] final_data = [] expense_data = [] for row in sheetData: split_row = row.split(",") final_data.append(split_row) for row in range(0, len(final_data)-1): for col in range(3, 30): loc = final_data[row][0] opsheet = final_data[row][1] rowNum = final_data[row][2] colName = colNames[col] str1 = "='\\\xxxxxxx\xxxx\xxx\[APR_" str2 = "_DATA.xlsm]" str3 = "\'!" str4 = "$" finalStr = str1 + loc + str2 + opsheet + str3 + colName + str4 + rowNum final_data[row][col] = "" with open("c:\users\xxxxx\new_expenses.csv", "wb") as f: fwriter = csv.writer(f) fwriter.writerows(final_data)
I'm a beginner in Python and I know this code is not up to the standards or a real way to code. But this code works fine and creates a CSV file.

Now the problem is this file is already created by someone else and I'm replicating it for other purposes with new data and everything. The existing file, created earlier, was only 30MB in size. But the file that is created in this method is 180MB and takes an enormous amount of time to open.

The original file is nothing but a single worksheet with 51,000 rows and 20 Columns each cell referencing to some cell in a workbook in a network location.

Any idea on how to reduce the size? or What is causing the huge?
Also as a beginner, any suggestions/feedback to improving the code is really appreciated. Thanks in advance!!
Reply


Possibly Related Threads…
Thread Author Replies Views Last Post
  How can I write formatted (i.e. bold, italic, change font size, etc.) text to a file? JohnJSal 13 35,366 May-20-2025, 12:26 PM
Last Post: hanmen9527
Question [redistribution] Reduce number + size of dependencies? Winfried 2 1,281 Jan-31-2025, 10:17 PM
Last Post: snippsat
  Changing client.get() method type based on size of data... dl0dth 1 1,273 Jan-02-2025, 08:30 PM
Last Post: dl0dth
  Python 3.13 : def setup size HansieB 6 2,367 Dec-02-2024, 11:12 AM
Last Post: buran
  IndexError: index 31 is out of bounds for axis 0 with size 31 YL789 1 1,718 Sep-21-2024, 09:46 AM
Last Post: Gribouillis
  File is not being created with 'w' argument CAD79 3 2,265 Mar-14-2024, 12:05 PM
Last Post: snippsat
  Python code for alignment and font size 1418 0 1,371 Jan-14-2024, 03:56 AM
Last Post: 1418
Question [PyMuPDF] Grab all strings of a given size? Winfried 3 2,135 Dec-26-2023, 07:39 AM
Last Post: Pedroski55
  howto get size of a ctk image? janeik 2 3,148 Oct-03-2023, 03:49 AM
Last Post: janeik
  IndexError: index 10 is out of bounds for axis 0 with size 10 Mehboob 11 16,081 Sep-14-2023, 06:54 AM
Last Post: Mehboob

User Panel Messages

Announcements
Announcement #1 8/1/2020
Announcement #2 8/2/2020
Announcement #3 8/6/2020
This forum uses Lukasz Tkacz MyBB addons.
Forum use Krzysztof "Supryk" Supryczynski addons.