Comparing Different OpenAI Models on Extracting Structured Information from PDF Documents


I was working on a problem where I needed to extract information from hotel tariff sheet PDF documents. These documents provide details on seasonal room rates, occupancy terms, and related supplements. They serve as standard reference material for travel agents, tour operators, and partners when contracting accommodations. Below is a screenshot of a synthetic document (similar to the original) that I created using ChatGPT.

For this use case I used OpenAI responses API. I tried extraction with gpt-4.1-mini, gpt-4o, gpt-4o-mini, gpt-5-nanoand gpt-5-mini models.

The PDFs were 2-5 pages long. I converted them to images and then used OpenAI’s responses API with structured output to extract the information.

 def extract_contract_from_pdf(self, pdf_bytes: bytes, model: str = "gpt-5-mini") -> tuple[ContractInformation, int]: try: images = [ { "type": "input_image", "image_url": f"data:image/png;base64,{self.image_to_base64(i)}" } for i in self.pdf_to_images(pdf_bytes)] response = self.client.responses.parse( model=model, input=[ { "role": "system", "content": """Extract the contract information from the PDF page images. Follow this two-step process: 1) First, carefully scan all pages and identify ALL room types mentioned in the contract 2) Then, for each room type you identified, extract all relevant details including pricing, occupancy, amenities, and any special conditions. Ensure you capture every room type - do not skip any.""" }, { "role": "user", "content": images } ], text_format=ContractInformation ) final_contract = response.output_parsed return final_contract, len(images)

ContractInformation is a Pydantic model. I added chain_of_thought to the Pydantic model classes to enforce chain-of-thought reasoning for non-reasoning models.

Below are my notes on how different models performed.

gpt-4o-mini

  • Processing time: 55.48 seconds
  • Token usage: 76,880 (Input: 74,494, Output: 2,386) ~= USD 0.01
  • Generated JSON was 454 lines long
  • Figured out currency correctly. Also, it didn’t get confused by currency and rate closer to each other.
  • It figured out period year from the context. Period date range only has date and month. It figured out year from the context.
  • Failed to extract all room types. It only extracted 6 room types.
  • Failed to extract both date ranges. Period A, D, and E have two date ranges. The extracted JSON data had only one date. It merged both the dates and created a single date range. For example, for Period A it created single date range Apr 1, 2026 to Apr 30, 2026. In reality it had two date ranges – Apr 1 to Apr 12 and Apr 21 to Apr 30
  • Extracted property name, address, and location. Duplicated Greece in both.
  • Azure Coastal Resort
  • Sunset Bay 1288, Greece
  • Greece

gpt-4o

  • Processing time: 208.37 seconds
  • Token usage: 11,638 (Input: 3,034, Output: 8,604) ~= USD 0.09
  • Generated JSON was 1606 lines long
  • Extracted all the 22 room types
  • Got the pricing and period date ranges wrong. For Period A, we have two date ranges: 01.04 to 12.04 and 21.04 to 30.04. The price for both is 210 Euro. The gpt-4o model extracted price 210 Euro for date range 01.04 to 21.04. It also got date ranges wrong for all the periods.
  • Extracted property name, address, and location.
  • Azure Coastal Resort
  • Sunset Bay 1288, Greece
  • Location was empty

JSON for one of the room type

 { "chain_of_thought": "I identified the 'Junior Suite, Pool View' and found the rates across different periods.", "name": "Junior Suite, Pool View", "rate_periods": [ { "rate": "210", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Converted 01.04 to 21.04.2026 to standard format.", "start_date": "2026-04-01", "end_date": "2026-04-21" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "218", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Converted 22.04 to 30.04.2026 to standard format.", "start_date": "2026-04-22", "end_date": "2026-04-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "233", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Converted 01.05 to 30.06.2026 to standard format.", "start_date": "2026-05-01", "end_date": "2026-06-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "239", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Converted 01.07 to 31.08.2026 to standard format.", "start_date": "2026-07-01", "end_date": "2026-08-31" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "269", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Converted 01.09 to 31.10.2026 to standard format.", "start_date": "2026-09-01", "end_date": "2026-10-31" } ], "release_period_days": 14, "minimum_stay_nights": 3 } ], "special_notes": null }

gpt-4.1-mini

  • Processing time: 119.55 seconds
  • Total Tokens: 14,541 (Input: 5,596, Output: 8,945) ~= USD 0.02
  • Generated JSON was 1826 lines long.
  • Extracted all the 22 room types.
  • Figured out currency correctly. Also, it didn’t get confused by currency and rate closer to each other.
  • It figured out period year from the context. Period date range only has date and month. It figured out year from the context.
  • Failed to extract both date ranges for all periods. Period A, D, and E have two date ranges. For different room types it made different mistakes.
  • For Period A, it extracted both date ranges correctly.
  • For Period D it extracted both date ranges but one of them was incorrect. Actual: Jul 1, 2026 to Jul 31, 2026 and Sep 1, 2026 to Sep 11, 2026. Extracted: Jul 1, 2026 to Aug 31, 2026 and Sep 1, 2026 to Sep 11, 2026
  • For Period E it only extracted one date range – Actual: Aug 1, 2026 to Aug 31, 2026 and Sep 12, 2026 to Oct 31, 2026. Extracted: Sep 12, 2026 to Oct 31, 2026
  • Extracted property name, address, and location.
  • Azure Coastal Resort
  • Address: Sunset Bay 1288, Greece
  • Location: Sunset Bay, Greece

Example JSON from one room type

 { "chain_of_thought": "Junior Suites have two variants: Pool View and Sea View, with max guests 3, standard occupancy 2, allotment 1 each. Pricing for periods A to E differ, with Sea View more expensive than Pool View.", "name": "Junior Suite, Pool View", "rate_periods": [ { "rate": "210", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period A 01.04–12.04 and 21.04–30.04", "start_date": "2026-04-01", "end_date": "2026-04-12" }, { "chain_of_thought": "and 21.04–30.04", "start_date": "2026-04-21", "end_date": "2026-04-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "218", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period B 13.04–20.04", "start_date": "2026-04-13", "end_date": "2026-04-20" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "233", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period C 01.05–30.06", "start_date": "2026-05-01", "end_date": "2026-06-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "239", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period D 01.07–31.08 and 01.09–11.09", "start_date": "2026-07-01", "end_date": "2026-08-31" }, { "chain_of_thought": "and 01.09–11.09", "start_date": "2026-09-01", "end_date": "2026-09-11" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "269", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period E 01.08–31.08 and 12.09–31.10", "start_date": "2026-09-12", "end_date": "2026-10-31" } ], "release_period_days": 14, "minimum_stay_nights": 3 } ], "special_notes": null },

gpt-4.1

  • Processing time: 355.77 seconds
  • Total Tokens: 15,747 (Input: 3,034, Output: 12,713) ~= USD 0.11
  • Generated JSON was 1841 lines long.
  • Extracted all the 22 room types.
  • Figured out currency correctly. Also, it didn’t get confused by currency and rate closer to each other.
  • It figured out period year from the context. Period date range only has date and month. It figured out year from the context.
  • Failed to extract both date ranges for all periods. Period A, D, and E have two date ranges. Below is just one example. For different room types it made different mistakes.
  • For Period A, it extracted both date ranges correctly.Actual: Apr 1, 2026 to Apr 12, 2026 and Apr 21, 2026 to Apr 30, 2026. Extracted: Apr 1, 2026 to Apr 21, 2026 and Apr 21, 2026 to Apr 30, 2026
  • For Period D it extracted both date ranges but second was incorrect. Actual: Jul 1, 2026 to Jul 31, 2026 and Sep 1, 2026 to Sep 11, 2026. Extracted: Jul 1, 2026 to Jul 31, 2026 and Sep 1, 2026 to Sep 10, 2026
  • For Period E it extracted both date ranges correctly- Actual: Aug 1, 2026 to Aug 31, 2026 and Sep 12, 2026 to Oct 31, 2026. Extracted: Aug 1, 2026 to Aug 31, 2026 and Sep 12, 2026 to Oct 31, 2026
  • Extracted property name, address, and location.
  • Azure Coastal Resort
  • Address: Sunset Bay 1288, Greece
  • Location: Greece

Below is JSON for one of the room types

 { "chain_of_thought": "Room type and all associated rates/dates extracted directly from rate table.", "name": "Junior Suite, Pool View", "rate_periods": [ { "rate": "210", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period A covers 01/04/2026 – 21/04/2026, 21/04/2026 – 30/04/2026", "start_date": "2026-04-01", "end_date": "2026-04-21" }, { "chain_of_thought": "Period A continued for extra days", "start_date": "2026-04-21", "end_date": "2026-04-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "218", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period B covers 13/04 – 20/04 (year is 2026 per context)", "start_date": "2026-04-13", "end_date": "2026-04-20" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "233", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period C covers 01/05/2026 – 30/06/2026", "start_date": "2026-05-01", "end_date": "2026-06-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "239", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period D covers 01/07/2026 – 31/07/2026 and 01/09/2026 – 10/09/2026", "start_date": "2026-07-01", "end_date": "2026-07-31" }, { "chain_of_thought": "Period D second half", "start_date": "2026-09-01", "end_date": "2026-09-10" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "269", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period E covers 01/08/2026 – 31/08/2026, 12/09/2026 – 31/10/2026", "start_date": "2026-08-01", "end_date": "2026-08-31" }, { "chain_of_thought": "Period E continued for remainder", "start_date": "2026-09-12", "end_date": "2026-10-31" } ], "release_period_days": 14, "minimum_stay_nights": 3 } ], "special_notes": null }

gpt-5-nano

  • Processing time: 79.16 seconds
  • Total Tokens: 18,987 (Input: 5,241, Output: 13,746, Reasoning: 6,272) ~= USD 0.01
  • Generated JSON was 1606 lines long.
  • Extracted all the 22 room types.
  • Figured out currency correctly. Also, it didn’t get confused by currency and rate closer to each other.
  • It figured out period year from the context. Period date range only has date and month. It figured out year from the context.
  • Failed to extract date range for all room types. Both start and end date were empty as shown in the JSON document.
  • Interestingly, chain_of_thought was empty as well.
  • Extracted property name, address, and location.
  • Azure Coastal Resort
  • Address: Sunset Bay 1288
  • Location: Greece

Below is JSON for one of the room types

 { "chain_of_thought": "", "name": "Junior Suite, Pool View", "rate_periods": [ { "rate": "210", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "", "start_date": null, "end_date": null } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "218", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "", "start_date": null, "end_date": null } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "233", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "", "start_date": null, "end_date": null } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "239", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "", "start_date": null, "end_date": null } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "269", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "", "start_date": null, "end_date": null } ], "release_period_days": 7, "minimum_stay_nights": 3 } ], "special_notes": "Max Guests: 3; Allotment: 1; Std. Occ.: 2; Amenities: Pool View. Rates are per suite/night on BB terms; standard occupancy 2." }

gpt-5-mini

  • Processing time: 257.63 seconds
  • Total Tokens: 21,178 (Input: 4,359, Output: 16,819, Reasoning: 2,432) ~= USD 0.03
  • Generated JSON was 1936 lines long.
  • Extracted all the 22 room types.
  • Figured out currency correctly. Also, it didn’t get confused by currency and rate closer to each other.
  • It figured out period year from the context. Period date range only has date and month. It figured out year from the context.
  • Correctly figured out different Period date ranges for all room types.
  • It had chain of thought filled
  • Extracted property name, address, and location.
  • Azure Coastal Resort
  • Address: Sunset Bay 1288, Greece
  • Location: Sunset Bay, Greece

Below is JSON for one of the room types

 { "chain_of_thought": "Junior Suite, Pool View row: recorded max guests, allotment, std occ and rates for periods A–E.", "name": "Junior Suite, Pool View", "rate_periods": [ { "rate": "210", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period A has two segments per header", "start_date": "2026-04-01", "end_date": "2026-04-12" }, { "chain_of_thought": "Period A second segment", "start_date": "2026-04-21", "end_date": "2026-04-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "218", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period B single segment", "start_date": "2026-04-13", "end_date": "2026-04-20" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "233", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period C single segment", "start_date": "2026-05-01", "end_date": "2026-06-30" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "239", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period D has two segments", "start_date": "2026-07-01", "end_date": "2026-07-31" }, { "chain_of_thought": "Period D second segment", "start_date": "2026-09-01", "end_date": "2026-09-11" } ], "release_period_days": 7, "minimum_stay_nights": 3 }, { "rate": "269", "currency": "EUR", "date_ranges": [ { "chain_of_thought": "Period E has two segments", "start_date": "2026-08-01", "end_date": "2026-08-31" }, { "chain_of_thought": "Period E second segment", "start_date": "2026-09-12", "end_date": "2026-10-31" } ], "release_period_days": 14, "minimum_stay_nights": 3 } ], "special_notes": "Max Guests: 3; Allotment: 1; Standard Occupancy: 2. Rate in EUR per suite per night on B&B terms. See general supplements/cancellation/payment notes." },

Conclusion

In my benchmarking, I found gpt-5-mini to be the best model for this task. It performed the task accurately, correctly handling the complex date range extractions that other models struggled with. While its latency was higher compared to the mini models, it was still faster than gpt-4o and gpt-4.1, and more cost-effective than both.

Here’s a comprehensive comparison table based on the evaluation results:

Model Performance Comparison

Aspectgpt-4o-minigpt-4ogpt-4.1-minigpt-4.1gpt-5-nanogpt-5-mini
Processing Time55.48s208.37s119.55s355.77s79.16s257.63s
Cost~$0.01~$0.09~$0.02~$0.11~$0.01~$0.03
JSON Output Size454 lines1,606 lines1,826 lines1,841 lines1,606 lines1,936 lines
Room Types Extracted6/22 ❌22/22 ✅22/22 ✅22/22 ✅22/22 ✅22/22 ✅
Date Range AccuracyFailed ❌Failed ❌Partial ⚠️Partial ⚠️Failed ❌Perfect ✅
Currency HandlingCorrect ✅Correct ✅Correct ✅Correct ✅Correct ✅Correct ✅
Year InferenceCorrect ✅Correct ✅Correct ✅Correct ✅Correct ✅Correct ✅
Chain of Thought QualityBasicGoodGoodGoodEmpty ❌Excellent ✅
Overall AccuracyPoorFairGoodGoodPoorExcellent

Discover more from Shekhar Gulati

Subscribe to get the latest posts sent to your email.

Leave a comment