Welcome to today’s daily kōrero!
Anyone can make the thread, first in first served. If you are here on a day and there’s no daily thread, feel free to create it!
Anyway, it’s just a chance to talk about your day, what you have planned, what you have done, etc.
So, how’s it going?
I do a lot of analytical work in my wider Team, and wanted to start adding Python to the arsenal, and have now just got access to Pycharm.
Wondering if there was anyone in a similar space (Data/Analytics), and was keen to hear some of the reasons you use Python, and how its incorporated into your workflow?
I’m a data engineer working mostly in Python and sql embedded in a data analytics team. Our main use cases are for ingestion pipelines (API sources, glue scripts, batch jobs in airflow and aws), and some work in pandas that doesn’t fit into our dbt sql models. I think it’s also nice for data exploration and sharing via jupyter/colab notebooks.
What are you thinking of using it for?
There’s a few different reasons that I’ve though about for now:
A lot of the data that we are working with is quite large, and it’s sometimes a struggle to work with it in Google Sheets / Excel (Unfortunately our workplace uses both for some reason)
I have some weekly reports that I’ve somehow ended up generating (Getting data via SQL, massaging the data, and presenting via a dashboard or sharing a spreadsheet.
For creating a repeatable set of calculations when someone asks for something (which I’m sort of doing via Powerquery or Google Apps Script)
I’m quite big on visualizations, so I want to give Matplotlib a go.
And I do of coding (Javascript & C++(Arduino)), and have always wanted to add Python to my list of skills, especially in recent times, as I begin to delve more into Data.
Those sound like perfect scenarios! One of the first projects that got me hooked on python was processing large csv files instead of opening them in excel and running visual basic on them.
If you haven’t already, you should check out duck db for working with your larger data sets, too. It’s pretty neat. https://duckdb.org/
I’ve had a brief look into duckdb, and not too sure if I’m interpreting it’s use case correctly, but does it basically allow you to use SQL within your Python to query your large datasets that you have locally?
That’s right. You can read in structured files and query them locally without having to load into a database. It’s nice in the case where you would rather write analytics sql, or want to convert between sql and pandas. It’s very quick to load and run files. It can connect to databases, too.
Oooh that sounds pretty promising - I’ve been struggling with how to handle quite large datasets when they don’t live within a Database.
Thank you for enlightening me! :) - I might have to send you some messages or the like later if I have any questions if that’s okay with you?
Sure thing!
Thank you :)
I started working with python last year and now it an every day tool for me. Some of it is data / analytics, taking graphql queries, using pandas to restructure etc.
I looked at pycharm for a while, but started out with VSCode; and then jumped to using a linux server & nvim instead.
Sounds pretty cool!
Did you have some programming experience before Python? - How long do you think it took for you to start being getting some “useful” outputs using Python?
Yes, but not really.
Yes - I started in the 80s typing in BASIC by hand into my C64, and did a really half assed visual basic course in tertiary study. But outside of that not really until I started playing around with esp32s and having to learn a bit of C until I found out that micropython ran on them and was way easier, plus you didn’t need to use the Arduino IDE.
Mucking about with sensors & ESP32 etc is actually a good way to intro yourself to it I think; especially as there’s a big community so loads of projects online to take inspiration from.
At work most of what I do is about manipulating data so we can take it from one system and merge it into another so I don’t have to do too much actual data analysis with it most of the time. Usually just simple maths like taking a rate from some prometheus data, figuring out what the actual configured speed on a circuit is & then converting it into a percentage so it can be visualised more easily.
All really basic stuff compared to what some folks get up to with python.
I’ve really been enjoying working with microcontrollers. Started with Arduino, and quickly moved onto ESP8266’s. I’ve got a couple of ESP32’s and have been meaning to start giving them a go, but have been procrastinating as usual lol.
We’ve actually got quite a big need for that as well - Moving Data from one system to another, and ensuring it’s in the right format etc. Might be able to get some real benefit out of that :)