The repository will be public, so drop the VIX series (Cboe data under a guessed license), the tweets (text by Twitter users) and TPC-H lineitem. NOAA's daily CO2 series (public domain, 18k rows past the row limit) and Apache Parquet's alltypes_tiny_pages (Apache-2.0, every primitive type and timestamps) replace them; the synthetic long-text sheet covers emoji.
- commit
- 1afda19a6874322390f40e9768202697725b9bca
- parent
- 35703a7dd3e0e5d1fe6350d31b15506669a10450
- author
- Dami <dami.etoile@gmail.com>
| M | scripts/ | 8 | ++++---- |
| M | scripts/ | 9 | ++++----- |
@@ -15,14 +15,14 @@
1515 # |--------------------------------|--------------------------------------------------|---------------------------------------------------|--------------------------|---------|------------------------------|1616 # | owid-energy-data.csv | wide CSV (130 cols) + rows past the 8192 limit | github owid/energy-data @7e387a1 | CC-BY-4.0 | 9229369 | 23377 x 130 |1717 # | airport-codes.csv | very long CSV (10x row limit), accented Latin | github datasets/airport-codes @08b5eef | ODC-PDDL-1.0 | 8809882 | 86134 x 13 |18-# | vix-daily.csv | numeric-heavy, just past row limit, CRLF | github datasets/finance-vix @3dbea23 | ODC-PDDL-1.0 | 482200 | 9278 x 5 |18+# | co2-ppm-daily.csv | numeric CSV past the row limit, dates | github datasets/co2-ppm-daily @ac476b2 | ODC-PDDL-1.0 (NOAA) | 347788 | 18304 x 2 |1919 # | country-codes.csv | Arabic/CJK/Cyrillic text, leading-zero codes | github datasets/country-codes @6a595f1 | ODC-PDDL-1.0 | 134003 | 249 x 56 |20-# | mayweather-mcgregor-tweets.csv | emoji, multi-line quoted cells, 18-digit ids | github fivethirtyeight/data @4c1ff5e | CC-BY-4.0 | 2386542 | 12118 x 7 |2120 # | FormulaEvalTestData_Copy.xlsx | XLSX formulas (1189 on sheet 1) + number formats | github apache/poi @942d95d test-data/spreadsheet | Apache-2.0 | 65011 | sheet 1 A2:AL1504 (788 rows) |2221 # | Chinook_Sqlite.sqlite | SQLite tables (PlaylistTrack past row limit) | github lerocha/chinook-database release v1.4.5 | MIT | 1067008 | 11 tables, max 8715 x 2 |23-# | lineitem-top10000.gzip.parquet | Parquet, GZIP pages, typed cols, past row limit | github duckdb/duckdb @cf0b14d data/parquet-testing| MIT | 292469 | 10000 x 16 |22+# | alltypes_tiny_pages.parquet | Parquet, every primitive type, tiny pages | github apache/parquet-testing @56653c4 | Apache-2.0 | 454233 | 7300 x 13 |2423 #25-# Total: about 22 MB.24+# Total: about 20 MB. Every file is openly licensed (CC-BY-4.0, public domain25+# dedications, Apache-2.0 or MIT); see the license column of the manifest.2626 #2727 # Usage: scripts/stress-data.sh (from anywhere)2828 # STRESS_DIR=/tmp/stress scripts/stress-data.sh@@ -1,9 +1,8 @@
11 name url sha256 bytes license shape purpose2-owid-energy-data.csv https://raw.githubusercontent.com/owid/energy-data/7e387a16f70a510e433f8aac7efeac6faa1e5059/owid-energy-data.csv 266f2e2baad7975351bc9bb4aa061d22b1da9fe4c47d51d2ac6071e01e171f76 9229369 CC-BY-4.0 (OWID; some upstream third-party sources keep their own terms) 23377 rows x 130 cols wide CSV (130 cols, mostly sparse floats) and long (2.85x the 8192-row limit)2+owid-energy-data.csv https://raw.githubusercontent.com/owid/energy-data/7e387a16f70a510e433f8aac7efeac6faa1e5059/owid-energy-data.csv 266f2e2baad7975351bc9bb4aa061d22b1da9fe4c47d51d2ac6071e01e171f76 9229369 CC-BY-4.0 (Our World in Data) 23377 rows x 130 cols wide CSV (130 cols, mostly sparse floats) and long (2.85x the 8192-row limit)33 airport-codes.csv https://raw.githubusercontent.com/datasets/airport-codes/08b5eefd8fedf5d9cc6e2136f86f448a38e1fcc3/data/airport-codes.csv 36a924db3b17a6f8ee55c242ed7f6fa3048e529d37fee6b3c4ad9527ea395778 8809882 ODC-PDDL-1.0 (datahub; data from OurAirports, public domain) 86134 rows x 13 cols very long CSV (10x the row limit, drop path and load speed); accented Latin names; quoted "lat, lon" cells4-vix-daily.csv https://raw.githubusercontent.com/datasets/finance-vix/3dbea23c70cc2afbeeec8ac232c9ec72ac26ac21/data/vix-daily.csv 570e05630c9a5e7350d0c3dbe3acc58e793cb5248181ab0e949c121cc082fb83 482200 ODC-PDDL-1.0 (datahub; source data from Cboe) 9278 rows x 5 cols numeric-heavy CSV just past the row limit (1086 rows dropped); ISO dates; CRLF line endings4+co2-ppm-daily.csv https://raw.githubusercontent.com/datasets/co2-ppm-daily/ac476b2abf2c1c5107eda109f5c5710145adec3a/data/co2-ppm-daily.csv 028668ad4dc7d4065f3fc26c41666f0a78163412c6d9971b4634035d073795ca 347788 ODC-PDDL-1.0 (datahub; NOAA data, public domain) 18304 rows x 2 cols numeric CSV past the row limit (10112 rows dropped); ISO dates become date cells55 country-codes.csv https://raw.githubusercontent.com/datasets/country-codes/6a595f1a6f10b3d00175fe67375da88f64f7f76b/data/country-codes.csv 67b009b529330b0a6043551189f43faa785c9c3cc0011ad2bdb4eac876356c43 134003 ODC-PDDL-1.0 (datahub) 249 rows x 56 cols unicode text: Arabic (RTL), CJK (double-width), Cyrillic names; codes with leading zeros; many text cols6-mayweather-mcgregor-tweets.csv https://raw.githubusercontent.com/fivethirtyeight/data/4c1ff5e3aef1816ae04af63218015066e186c147/mayweather-mcgregor/tweets.csv 6b4e965637075b9f983898989fb16ab2b56325b15b6404b3d8c7c67ed045a89f 2386542 CC-BY-4.0 (FiveThirtyEight data repo) 12118 rows x 7 cols text-heavy: emoji (ZWJ, VS16, flags), Arabic/Cyrillic, 1774 multi-line quoted cells, 18-digit ids (float precision), past the row limit7-FormulaEvalTestData_Copy.xlsx https://raw.githubusercontent.com/apache/poi/942d95d85b15d0dfdb3bc9ba1b4f273f277757c8/test-data/spreadsheet/FormulaEvalTestData_Copy.xlsx 01b21647d53501e8ec10f4eb522945d41ea3c8d1fdd57f8b0142ec34884c33eb 65011 Apache-2.0 (Apache POI) first sheet EverythingTests A2:AL1504, 788 rows, 4262 cells, 1189 formulas; 4 sheets; 7 custom number formats real XLSX with many formulas (wide function coverage, some unsupported) and number formats (currency with [Red], dates, fixed decimals, TRUE/FALSE)6+FormulaEvalTestData_Copy.xlsx https://raw.githubusercontent.com/apache/poi/942d95d85b15d0dfdb3bc9ba1b4f273f277757c8/test-data/spreadsheet/FormulaEvalTestData_Copy.xlsx 01b21647d53501e8ec10f4eb522945d41ea3c8d1fdd57f8b0142ec34884c33eb 65011 Apache-2.0 (Apache POI test data) first sheet EverythingTests A2:AL1504, 788 rows, 4262 cells, 1189 formulas; 4 sheets; 7 custom number formats real XLSX with many formulas (wide function coverage, some unsupported) and number formats (currency with [Red], dates, fixed decimals, TRUE/FALSE)87 Chinook_Sqlite.sqlite https://github.com/lerocha/chinook-database/releases/download/v1.4.5/Chinook_Sqlite.sqlite bdf635be69850bd3be09c9a2dbeef7ddfb80036bd3ef3381383cd03b61e4a61a 1067008 MIT (Chinook Database, Luis Rocha) 11 tables; Track 3503x9, PlaylistTrack 8715x2, InvoiceLine 2240, Customer 59x13, Invoice 412x9 SQLite table import; table picker; PlaylistTrack is past the row limit; accented names, NULLs, numeric prices9-lineitem-top10000.gzip.parquet https://raw.githubusercontent.com/duckdb/duckdb/cf0b14db3edaf39f6051ba2790df66d1680f687e/data/parquet-testing/lineitem-top10000.gzip.parquet b90d102aa05c4cde2609a1fef70153b12b2d86f2590922bc6734352d460fa573 292469 MIT (DuckDB repo; TPC-H dbgen output) 10000 rows x 16 cols Parquet with GZIP pages, bigint/int/double/varchar columns, past the row limit8+alltypes_tiny_pages.parquet https://raw.githubusercontent.com/apache/parquet-testing/56653c437c8092f704a092d0d1d4e600124cd49f/data/alltypes_tiny_pages.parquet f7a7678a53bfdb434d9a51f7f42a71365eae807b3f8e16bfcad67cd623748228 454233 Apache-2.0 (Apache Parquet test data) 7300 rows x 13 cols Parquet with every primitive type, timestamps and many tiny pages; under the row limit