I'm spending way too much time manually downloading interval data from utility portals. TVA's system here in Knoxville is particularly clunky - takes 15 minutes per account to export a month of data. Has anyone found tools or methods to automate this process? I've got about 50 commercial accounts that need monthly analysis.
Anyone using automated interval data downloads?
Gary, I feel your pain. Duke Energy's portal here in Charlotte is just as bad. I've been working with a programmer to build a web scraper but it's tricky because most utilities require login credentials. The legal implications worry me too - not sure if automated downloads violate their terms of service.
There are some third-party services that do this legally. I've been testing UtilityAPI for LG&E accounts here in Louisville. They have agreements with utilities to pull data automatically. Costs about $3 per meter per month but saves me probably 10 hours of manual work monthly. Worth every penny.
Pete, that's interesting about UtilityAPI. Do they work with TVA? $3 per meter is definitely reasonable if it eliminates the manual downloads. How's the data quality? Any missing intervals or formatting issues?
Data quality has been solid Gary. They normalize everything to standard CSV format which is nice. I'm not sure about TVA coverage - you'd need to check their utility list. They definitely cover most of the major IOUs. The automated alerts when new data is available are handy too.
I've been using Selenium WebDriver to automate TEP downloads here in Tucson. It's a bit technical to set up but once running it handles the login and download process automatically. Free solution if you're comfortable with some basic Python programming. Happy to share the script if anyone's interested.
Omar, I'd love to see your Python script. I have some programming background and the free approach appeals to me. How often do the utilities change their portal interfaces and break your automation? That's been my main concern with scraping approaches.
David, TEP has only changed their interface once in the past year and it was just minor CSS updates. Took about 30 minutes to fix the script. I build in error handling so if something breaks, it emails me rather than failing silently. The time investment upfront pays off quickly.
This thread has been incredibly helpful. I'm going to look into both UtilityAPI and Omar's Python approach. The manual downloading is killing my productivity. Anyone have experience with other third-party data services like WattTime or EnergyCAP?
Gary, I used EnergyCAP for OG&E accounts here in Oklahoma City for about six months. Good data quality but expensive - around $8 per meter monthly. Switched to manual downloads to save costs but now I'm rethinking that decision after reading this thread.
Susan, at $8 per meter EnergyCAP definitely adds up fast. The Python approach has zero ongoing costs once you get it working. I've sent my script to Pete and David - if others want it just PM me. Always happy to help fellow auditors automate the tedious stuff.