Tag jupyter

PySpark and Jupyter Notebook

There's a lot of crap advice about getting jupyter notebooks to play nicely with pyspark. I guess things have changed a lot over the last couple of years, but here's how I have things. I use conda for my python envs, but I doubt that...

Scrubbing of (poor) data.

I have sensors - a great many - which report numbers daily. There's a long piepline and many processes between these sensorss and the ~daily reports I get, and sometimes weird spikes and dips happen in the numbers: often fixed the very...