Welcome to today’s daily kōrero!
Anyone can make the thread, first in first served. If you are here on a day and there’s no daily thread, feel free to create it!
Anyway, it’s just a chance to talk about your day, what you have planned, what you have done, etc.
So, how’s it going?
Those sound like perfect scenarios! One of the first projects that got me hooked on python was processing large csv files instead of opening them in excel and running visual basic on them.
If you haven’t already, you should check out duck db for working with your larger data sets, too. It’s pretty neat. https://duckdb.org/
I’ve had a brief look into duckdb, and not too sure if I’m interpreting it’s use case correctly, but does it basically allow you to use SQL within your Python to query your large datasets that you have locally?
That’s right. You can read in structured files and query them locally without having to load into a database. It’s nice in the case where you would rather write analytics sql, or want to convert between sql and pandas. It’s very quick to load and run files. It can connect to databases, too.
Oooh that sounds pretty promising - I’ve been struggling with how to handle quite large datasets when they don’t live within a Database.
Thank you for enlightening me! :) - I might have to send you some messages or the like later if I have any questions if that’s okay with you?
Sure thing!
Thank you :)