Open Journalism, Open Data.
I love this phrase, and if you know me, it tracks. “You might be considered a strange guy,” is one of the highest compliments I’ve ever received. [2015 audio clip on YouTube] I try to track everything happening in Hamilton’s civic affairs. My personal server is littered with scripts, kanboards, and spreadsheets that I use to track hundreds of data points.
Finally, thanks to advancements in large-language models, coding assistants, and Python scripting languages; I am able to modify my messy data into open data that anyone can access. (Aside: you all know that person with the messy desk, the person who somehow knows where everything is, the person who if you clean their desk and properly organize things would be lost. That’s kinda me.)
In the coming weeks, I’ll be releasing the following data sets as both open data and web applications:
- Hamilton City Council attendance records: a complete record of all members since 2022, including council roll call data;
- Voting records: every vote, every committee, agency, and board, since 2022;
- A development and planning tracker, which will include Ontario Land Tribunal matters; and
- more to follow.
First Data Set Launched: Hamilton Building Permits (2011 to Present)
The first data set to launch uses sixteen years of Statistics Canada monthly “Building Activity” reports for the City of Hamilton to create a searchable database.
The TPR Hamilton Open Data Building Permits site allows you to search and find information regarding permits issued by the City of Hamilton during the past 16 years.
I’ve kept the monthly records since 2011, 197 individual PDFs of messy data. (The City of Hamilton’s website includes copies since September 2021 here.)
The PDFs are “messy,” meaning each has different formatting, errors, and inconsistencies from years of inconsistent input conventions. Street names are missing, naming conventions are inconsistent, and on top of this, some of the records do not state the former municipality for the address. There are King streets, Main streets, John streets, Highway 6 and Highway 8 addresses that are shared between Ancaster, Dundas, Flamborough, Glanbrook, Hamilton, and Stoney Creek.
Until recently, it was too time-consuming to scrape page-by-page. This changed with recent LLMs, commonly referred to as AI. Coding assistants are capable of writing and modifying Python scraping scripts for each PDF page.
Following data scraping, there were approximately 120,000 individual permit records. The addresses were cross-referenced against the City’s master table of street addresses, and thousands of street names were standardized to the proper formatting. This left approximately 5,000 unresolved records.
In the coming weeks, I’ll publish a detailed explainer of the processes I used to resolve those 5,000 records. After data cross-references, clean-up of historical survey plans and former street names, and even some zoning verifications against the permit type in the Statistics Canada database, I was left with three records that required me to contact the City for the original permit records.
I look forward to sharing the work.
Browse, Search, Download, and Provide Your Feedback
I invite you to browse the database. You can search any address, view the latest month’s data, sort by the most active and highest-value addresses, category of permit, or municipality, neighbourhood, or ward, and more. A simple click allows you to see past permit values adjusted for inflation.
The full data explorer feature allows you to filter and download the raw data in CSV format. You can even download all 113,433 records in GeoJSON format.
True open data!
I have only one request. Send me your feedback so I can make improvements.
Sincerely,
Joey Coleman
Production Details
v. 1.0.0
Published: July 19, 2026
Last updated: July 19, 2026
Author: Joey Coleman
Update Record
v. 1.0.0 original version
Discussion
1 commentThanks for the transparency!