scripts/stress-data.sh downloads the files listed in stress-data.tsv into .deps/stress (gitignored), verifies their SHA-256 and skips files already present. Eight datasets, about 22 MB, all pinned to a commit or release: OWID energy (130 columns, 23k rows, CC-BY-4.0), airport codes (86k rows, PDDL), VIX daily (numbers, PDDL), country codes (Arabic, CJK, Cyrillic, PDDL), FiveThirtyEight tweets (emoji, multi-line cells, CC-BY-4.0), Apache POI's formula test workbook (1189 formulas, Apache-2.0), Chinook (SQLite, MIT) and TPC-H lineitem from DuckDB's Parquet test data (MIT).
- commit
- be396942492b6e64bf8219c6c4443577e0191ded
- parent
- 0f86bf4d0d5f5b48b53e69c670a38a6da60c4c0f
- author
- Dami <dami.etoile@gmail.com>
| A | scripts/ | 101 | ++++++++++++++++++++++++++++++ |
| A | scripts/ | 9 | +++ |
@@ -0,0 +1,101 @@
1+2+# stress-data.sh - fetch real public datasets for stress-testing 012's importers.3+#4+# Downloads every file listed in scripts/stress-data.tsv into .deps/stress/5+# (override with STRESS_DIR), verifies SHA-256, and is idempotent: files that6+# are already present with the right checksum are left alone, files with a7+# wrong checksum are re-downloaded, and a download that still does not match8+# fails loudly. All URLs are pinned to a commit hash or a versioned release.9+#10+# The manifest (scripts/stress-data.tsv, tab-separated, one header row:11+# name, url, sha256, bytes, license, shape, purpose) is the single source of12+# truth; Go code can read it too. The table below is a summary of it.13+#14+# | file | exercises | source (pinned) | license | bytes | rows x cols |15+# |--------------------------------|--------------------------------------------------|---------------------------------------------------|--------------------------|---------|------------------------------|16+# | owid-energy-data.csv | wide CSV (130 cols) + rows past the 8192 limit | github owid/energy-data @7e387a1 | CC-BY-4.0 | 9229369 | 23377 x 130 |17+# | airport-codes.csv | very long CSV (10x row limit), accented Latin | github datasets/airport-codes @08b5eef | ODC-PDDL-1.0 | 8809882 | 86134 x 13 |18+# | vix-daily.csv | numeric-heavy, just past row limit, CRLF | github datasets/finance-vix @3dbea23 | ODC-PDDL-1.0 | 482200 | 9278 x 5 |19+# | country-codes.csv | Arabic/CJK/Cyrillic text, leading-zero codes | github datasets/country-codes @6a595f1 | ODC-PDDL-1.0 | 134003 | 249 x 56 |20+# | mayweather-mcgregor-tweets.csv | emoji, multi-line quoted cells, 18-digit ids | github fivethirtyeight/data @4c1ff5e | CC-BY-4.0 | 2386542 | 12118 x 7 |21+# | FormulaEvalTestData_Copy.xlsx | XLSX formulas (1189 on sheet 1) + number formats | github apache/poi @942d95d test-data/spreadsheet | Apache-2.0 | 65011 | sheet 1 A2:AL1504 (788 rows) |22+# | Chinook_Sqlite.sqlite | SQLite tables (PlaylistTrack past row limit) | github lerocha/chinook-database release v1.4.5 | MIT | 1067008 | 11 tables, max 8715 x 2 |23+# | lineitem-top10000.gzip.parquet | Parquet, GZIP pages, typed cols, past row limit | github duckdb/duckdb @cf0b14d data/parquet-testing| MIT | 292469 | 10000 x 16 |24+#25+# Total: about 22 MB.26+#27+# Usage: scripts/stress-data.sh (from anywhere)28+# STRESS_DIR=/tmp/stress scripts/stress-data.sh29+30+set -eu31+32+script_dir=$(cd "$(dirname "$0")" && pwd)33+repo_root=$(cd "$script_dir/.." && pwd)34+manifest="$script_dir/stress-data.tsv"35+dest=${STRESS_DIR:-"$repo_root/.deps/stress"}36+37+if [ ! -f "$manifest" ]; then38+ echo "stress-data: manifest not found: $manifest" >&239+ exit 140+fi41+42+if command -v shasum >/dev/null 2>&1; then43+ sha256() { shasum -a 256 "$1" | awk '{print $1}'; }44+elif command -v sha256sum >/dev/null 2>&1; then45+ sha256() { sha256sum "$1" | awk '{print $1}'; }46+else47+ echo "stress-data: need shasum or sha256sum" >&248+ exit 149+fi50+51+filesize() { wc -c <"$1" | tr -d ' '; }52+53+mkdir -p "$dest"54+55+failed=056+first=157+while IFS=$'\t' read -r name url want bytes _license _shape _purpose; do58+ if [ "$first" = 1 ]; then # header row59+ first=060+ continue61+ fi62+ [ -n "$name" ] || continue63+ path="$dest/$name"64+65+ if [ -f "$path" ] && [ "$(sha256 "$path")" = "$want" ]; then66+ printf '%-32s %10s ok\n' "$name" "$(filesize "$path")"67+ continue68+ fi69+70+ if [ -f "$path" ]; then71+ echo "stress-data: $name: checksum mismatch on disk, re-downloading" >&272+ fi73+74+ tmp="$path.part"75+ rm -f "$tmp"76+ if ! curl -fL --retry 3 --silent --show-error -o "$tmp" "$url"; then77+ rm -f "$tmp"78+ echo "stress-data: ERROR: $name: download failed: $url" >&279+ failed=180+ continue81+ fi82+83+ got=$(sha256 "$tmp")84+ if [ "$got" != "$want" ]; then85+ echo "stress-data: ERROR: $name: SHA-256 mismatch after download" >&286+ echo " url: $url" >&287+ echo " expected: $want ($bytes bytes)" >&288+ echo " got: $got ($(filesize "$tmp") bytes)" >&289+ echo " the upstream file changed or the download is corrupt; kept as $tmp" >&290+ failed=191+ continue92+ fi93+94+ mv -f "$tmp" "$path"95+ printf '%-32s %10s downloaded\n' "$name" "$(filesize "$path")"96+done <"$manifest"97+98+if [ "$failed" != 0 ]; then99+ echo "stress-data: FAILED: one or more files could not be fetched and verified (see above)" >&2100+ exit 1101+fi@@ -0,0 +1,9 @@
1+name url sha256 bytes license shape purpose2+owid-energy-data.csv https://raw.githubusercontent.com/owid/energy-data/7e387a16f70a510e433f8aac7efeac6faa1e5059/owid-energy-data.csv 266f2e2baad7975351bc9bb4aa061d22b1da9fe4c47d51d2ac6071e01e171f76 9229369 CC-BY-4.0 (OWID; some upstream third-party sources keep their own terms) 23377 rows x 130 cols wide CSV (130 cols, mostly sparse floats) and long (2.85x the 8192-row limit)3+airport-codes.csv https://raw.githubusercontent.com/datasets/airport-codes/08b5eefd8fedf5d9cc6e2136f86f448a38e1fcc3/data/airport-codes.csv 36a924db3b17a6f8ee55c242ed7f6fa3048e529d37fee6b3c4ad9527ea395778 8809882 ODC-PDDL-1.0 (datahub; data from OurAirports, public domain) 86134 rows x 13 cols very long CSV (10x the row limit, drop path and load speed); accented Latin names; quoted "lat, lon" cells4+vix-daily.csv https://raw.githubusercontent.com/datasets/finance-vix/3dbea23c70cc2afbeeec8ac232c9ec72ac26ac21/data/vix-daily.csv 570e05630c9a5e7350d0c3dbe3acc58e793cb5248181ab0e949c121cc082fb83 482200 ODC-PDDL-1.0 (datahub; source data from Cboe) 9278 rows x 5 cols numeric-heavy CSV just past the row limit (1086 rows dropped); ISO dates; CRLF line endings5+country-codes.csv https://raw.githubusercontent.com/datasets/country-codes/6a595f1a6f10b3d00175fe67375da88f64f7f76b/data/country-codes.csv 67b009b529330b0a6043551189f43faa785c9c3cc0011ad2bdb4eac876356c43 134003 ODC-PDDL-1.0 (datahub) 249 rows x 56 cols unicode text: Arabic (RTL), CJK (double-width), Cyrillic names; codes with leading zeros; many text cols6+mayweather-mcgregor-tweets.csv https://raw.githubusercontent.com/fivethirtyeight/data/4c1ff5e3aef1816ae04af63218015066e186c147/mayweather-mcgregor/tweets.csv 6b4e965637075b9f983898989fb16ab2b56325b15b6404b3d8c7c67ed045a89f 2386542 CC-BY-4.0 (FiveThirtyEight data repo) 12118 rows x 7 cols text-heavy: emoji (ZWJ, VS16, flags), Arabic/Cyrillic, 1774 multi-line quoted cells, 18-digit ids (float precision), past the row limit7+FormulaEvalTestData_Copy.xlsx https://raw.githubusercontent.com/apache/poi/942d95d85b15d0dfdb3bc9ba1b4f273f277757c8/test-data/spreadsheet/FormulaEvalTestData_Copy.xlsx 01b21647d53501e8ec10f4eb522945d41ea3c8d1fdd57f8b0142ec34884c33eb 65011 Apache-2.0 (Apache POI) first sheet EverythingTests A2:AL1504, 788 rows, 4262 cells, 1189 formulas; 4 sheets; 7 custom number formats real XLSX with many formulas (wide function coverage, some unsupported) and number formats (currency with [Red], dates, fixed decimals, TRUE/FALSE)8+Chinook_Sqlite.sqlite https://github.com/lerocha/chinook-database/releases/download/v1.4.5/Chinook_Sqlite.sqlite bdf635be69850bd3be09c9a2dbeef7ddfb80036bd3ef3381383cd03b61e4a61a 1067008 MIT (Chinook Database, Luis Rocha) 11 tables; Track 3503x9, PlaylistTrack 8715x2, InvoiceLine 2240, Customer 59x13, Invoice 412x9 SQLite table import; table picker; PlaylistTrack is past the row limit; accented names, NULLs, numeric prices9+lineitem-top10000.gzip.parquet https://raw.githubusercontent.com/duckdb/duckdb/cf0b14db3edaf39f6051ba2790df66d1680f687e/data/parquet-testing/lineitem-top10000.gzip.parquet b90d102aa05c4cde2609a1fef70153b12b2d86f2590922bc6734352d460fa573 292469 MIT (DuckDB repo; TPC-H dbgen output) 10000 rows x 16 cols Parquet with GZIP pages, bigint/int/double/varchar columns, past the row limit