Chapter 2 · Practice 3 of 4
Check data types when you load data, especially strings and dates
Why it matters
Type surprises change results silently: a numeric column read as text sorts "100" before "20", and dates parsed in the wrong unit shift every calculation.
pandas 3.0 changed two defaults worth knowing:
| Change in pandas 3.0 | What to watch |
|---|---|
String data now gets a dedicated str dtype instead of object | Code that checks for object dtype may stop matching |
| Parsed datetimes default to microsecond resolution, falling back to nanoseconds when needed | Casting datetimes to integers gives values 1000 times smaller than before |
After loading, print the schema (df.dtypes in pandas, df.schema in Polars) and compare it with what you expect. Parse dates explicitly with a format and time zone rather than relying on inference.
Avoid
- Converting datetimes to integers with astype("int64") without fixing the unit first