View the previous post at https://friendlyuser.github.io/posts/stonks/scrap_from_sec_with_python
This code fetches 13F filings for a list of companies from the Securities and Exchange Commission's (SEC) Electronic Data Gathering, Analysis, and Retrieval (EDGAR) system. 13F filings are quarterly reports that detail the holdings of investment managers with at least $100 million in assets. The filings function from the secedgar library is used to retrieve these 13F filings, and the get_company_of_interest function acts as a filter to only return the 13F filings for the companies specified in the company_list variable. This list is read from the holdings_list.txt file. The fetch_filings function uses the save method of the combo_filings object to save the retrieved 13F filings to the temp folder. The get_year_quarter_from_path function extracts the year and quarter from the path of a file and returns them as a tuple of strings.
from secedgar import filings, FilingType from datetime import date import glob import matplotlib.pyplot as plt import seaborn as sns import re import pandas as pd import json from typing import List from parser import DocParser # read company list from holdings_list.txt with open("holdings_list.txt", "r") as f: holdings_list = f.read().splitlines() company_list = holdings_list def get_company_of_interest(filing_entry): if "13F" in filing_entry.form_type and filing_entry.company_name.lower() in (name.lower() for name in company_list): company_list.append(filing_entry.company_name) return True return False def fetch_filings(): # 13F filings for Apple (ticker "aapl") # read list from holdings_list.txt combo_filings = filings(start_date=date(2022, 1, 1), end_date=date(2022, 12, 20), # filing_type=FilingType.FILING_13F, user_agent="Your name <dlcoding20@gmail.com>", entry_filter=get_company_of_interest, rate_limit=5 ) # map folder to year and quarter combo_filings.save("temp") def get_year_quarter_from_path(path: str): """ Get year and quarter from path """ # get all numbers from path using regex year, quarter = re.findall(r'\d+', path)[0:2] return year, quarter This code appears to be parsing 13F filing documents and outputting the results to a file in markdown format. The parse_filings function takes a dictionary of data (data) as input and retrieves a company's CIK number (a unique identifier assigned by the SEC to publicly traded companies) from the cik field in the data dictionary. If the cik field is not present, it defaults to "1649339". It also retrieves the desired output filename from the filename field in the data dictionary, defaulting to "burry" if the field is not present. The function then uses the glob library to find all the text files under the temp folder that match the specified CIK number and stores their filenames in the filename variable. It uses the DocParser class to parse each file and filters the resulting documents for those of type "INFORMATION_TABLE". It then adds the year and quarter of each document to its data frame and stores the resulting document in the final_docs list. If the final_docs list has four elements, the loop breaks. Finally, the function sorts the elements of final_docs by quarter and calls the output_to_md function to output the data to a markdown file. The metadata dictionary stores some metadata about the 13F filing that is included in the output file.
def parse_filings(data: dict = {}): """ Parse filings """ cik = data.get("cik", "1649339") output_name = data.get("filename", "burry") # find files under temp/2022/QTR{1,2,3,4}/*.txt with glob final_docs = [] for filename in glob.iglob(f'temp/2022/QTR*/{cik}/*.txt', recursive=True): documents = DocParser(filename, "13F").parse() # filter for INFORMATION_TABLE for document in documents: if document["type"] == "INFORMATION_TABLE": document["filename"] = filename # add year and quarter to document year, quarter = get_year_quarter_from_path(filename) temp_df = document["df"] # add year quarter to df temp_df["year"] = year temp_df["quarter"] = quarter final_docs.append(document) if len(final_docs) == 4: break # sort by quarter final_docs = sorted(final_docs, key=lambda x: x["df"]["quarter"].iloc[0]) metadata ={ "filename": f"{output_name}.md", "company_name": data.get("outputLabel", "Burry"), "category": "13F", "date": "2022-12-21", "start_date": "2022-01-01", "end_date": "2022-12-20", "cik": cik, } # eventually parse all this from a metadata yaml file or json file output_to_md(final_docs, metadata) This code appears to be processing and outputting data from 13F filing documents. The output_to_md function takes a list of data frames (final_docs) and a dictionary of metadata (metadata) as input. It first combines all the data frames in final_docs into a single data frame called combined_df. It then groups combined_df by the cusip column and stores the resulting data frame in grouped_df. It then extracts the last two quarters of data from final_docs and stores them in a data frame called last_two_quarters. It then converts the value column in last_two_quarters to numeric values and pivots the data frame by nameOfIssuer and quarter, storing the resulting data frame in pivot_df. It then plots pivot_df as a bar chart and saves the chart to a file. It then merges the last two quarters of data from final_docs using the pd.merge function and stores the resulting data frame in last_two_quarter_diff.
The function then opens a file with the name specified in the filename field of the metadata dictionary and writes some metadata and the pivoted data frame to it in markdown format. It also writes the data from each data frame in final_docs to the file, grouped by cusip. Finally, it writes the merged data frame (last_two_quarter_diff) to the file in markdown format.
def output_to_md(final_docs: List[pd.DataFrame], metadata: dict): """ Output dataframe to markdown file """ combined_df = None # iterate across final_docs and append them all for doc in final_docs: if combined_df is None: combined_df = doc["df"] else: # pandas.concat combined_df = pd.concat([combined_df, doc["df"]]) # combine df then group by cusip grouped_df = combined_df.groupby("cusip") # get 2nd last element in list last_quarter = final_docs[-2] current_quarter = final_docs[-1] filename = metadata["filename"] csv_filename = filename.replace(".md", ".csv") # write to csv combined_df.to_csv(csv_filename, index=False) merge_df = pd.merge(last_quarter["df"], current_quarter["df"], how="outer") # quarterly_df = combined_df.groupby("quarter") # append last two quarters last_two_quarters = pd.concat([last_quarter["df"], current_quarter["df"]]) last_two_quarters = last_two_quarters[["quarter", "nameOfIssuer", "value"]] # plot ax grouped by quarter # make value numeric # use .loc[row_indexer, col_indexer] last_two_quarters.loc[:, "value"] = pd.to_numeric(last_two_quarters["value"]) png_filename = filename.replace(".md", ".png") # make pivot table pivot_df = last_two_quarters.pivot(index="nameOfIssuer", columns="quarter", values="value") # plot pivot table # first color red, second color blue ax = pivot_df.plot(kind="bar", color=["blue", "green"]) # make image big enough for labels ax.figure.savefig(png_filename, bbox_inches="tight") # quarter diff combined last_two_quarter_diff = pd.merge(last_quarter["df"], current_quarter["df"], how="outer", indicator=True) with open(filename, "w") as f: f.write(f"Title: {metadata['company_name']} - {metadata['cik']} \n") f.write(f"Date: {metadata['date']} \n") f.write(f"Category: {metadata['category']} \n") # merge df f.write("## Quarterly holdings \n\n") # write png_filename f.write(f" \n\n") f.write("uses last two quarters to compare \n\n") f.write(f"## All tables \n\n") # write all tables reduced_df = combined_df[["nameOfIssuer", "value", "sshPrnamt", "sshPrnamtType", "year", "quarter"]] # if combined_df has putCall column if "putCall" in combined_df.columns: reduced_df["putCall"] = combined_df["putCall"] f.write(reduced_df.to_markdown(index=False)) f.write("\n\n") # write header f.write("## Tables by cusip \n\n") for cusip, group in grouped_df: f.write("\n \n") f.write(f"### {cusip} \n\n") simple_group = group[["nameOfIssuer", "value", "sshPrnamt", "sshPrnamtType", "year", "quarter"]] if "putCall" in group.columns: simple_group["putCall"] = group["putCall"] f.write(simple_group.to_markdown(index=False)) f.write("\n \n") f.write("\n\n") To view the sample output, please go to the following link:
13F.grandfleet.eu.org
It is generated by pelican, a static site generator written in python. The source code for the site is available at the following link:
https://github.com/FriendlyUser/13F-sec
Pelican is a static site generator written in Python. It is used to generate static HTML files for a website from templates and content written in a markup language such as Markdown or reStructuredText. Pelican is designed to be flexible, allowing users to customize the look and feel of their website and to extend its functionality through plugins. It also integrates with popular tools such as Jinja2 templates and the Pygments syntax highlighter. Pelican is a popular choice for bloggers, documentation authors, and other users who want to create a simple, fast-loading website without the need for a dynamic backend.
Top comments (0)