High-Flying Birds¶
This chapter uses VisiData to answer this question: Which birds that are struck, on average, at the highest altitudes?
In the process, we’ll cover the following skills:
- Assigning “types” to columns
- Creating new columns
- Using frequency tables to filter sheets
- Calculating aggregate statistics
Note
If you haven’t already, download faa-wildlife-strikes.csv, a dataset of all aircraft-wildlife collisions reported to the Federal Aviation Adminsitration between 2010 and mid-2016.
Open the wildlife-strikes dataset in VisiData¶
Let’s start from the very beginning.
Run this command in your terminal:
vd faa-wildlife-strikes.csv
If it worked, you should see something like this:
OPERATOR | ATYPE | INCIDENT_DATE | STATE | AIRPORT | PHASE_> BUSINESS | PA-28 | 05/22/15 00:00:00 | FL | VERO BEACH MUNICIP…| APPROA… BUSINESS | BE-1900 | 06/18/15 00:00:00 | AK | KENAI MUNICIPAL AR…| APPROA… BUSINESS | PA-46 MALIBU | 09/20/15 00:00:00 | TX | DAVID WAYNE HOOKS …| DELTA AIR LINES | B-717-200 | 11/07/15 00:00:00 | MO | LAMBERT-ST LOUIS I…| APPROA… BUSINESS | BE-90 KING | 12/17/15 00:00:00 | FL | POMPANO BEACH AIRP…| LANDIN… DELTA AIR LINES | B-757 | 07/17/15 00:00:00 | VI | HENRY E ROHLSEN AR…| DELTA AIR LINES | B-717-200 | 08/02/15 00:00:00 | TX | SAN ANTONIO INTL | APPROA… BUSINESS | C-414 | 08/03/15 00:00:00 | TX | LONE STAR EXECUTIV…| DEPART… ALLEGIANT AIR | MD-80 | 09/02/15 00:00:00 | FL | TAMPA INTL | APPROA… TRANS STATES AIRLI…| EMB-145 | 09/07/15 00:00:00 | MO | LAMBERT-ST LOUIS I…| APPROA… BUSINESS | C-172 | 11/28/15 00:00:00 | FL | OPA-LOCKA EXECUTIV…| APPROA… GOVERNMENT | EC120 | 12/08/15 00:00:00 | CA | NORMAN Y. MINETA S…| AMERICAN AIRLINES | A-321 | 05/06/15 00:00:00 | FL | FORT LAUDERDALE/HO…| APPROA… EXPRESSJET AIRLINES| CRJ100/200 | 05/06/15 00:00:00 | AR | FORT SMITH REGIONA…| CLIMB MESA AIRLINES | CRJ900 | 05/08/15 00:00:00 | AR | BILL AND HILLARY …| LANDIN… BUSINESS | HELICOPTER | 05/06/15 00:00:00 | | UNKNOWN | En Rou… DELTA AIR LINES | A-320 | 05/07/15 00:00:00 | CA | METRO OAKLAND INTL | DELTA AIR LINES | A-320 | 05/08/15 00:00:00 | UT | SALT LAKE CITY INTL| LUFTHANSA | A-380 | 05/10/15 00:00:00 | TX | GEORGE BUSH INTERC…| CLIMB BUSINESS | C-172 | 05/08/15 00:00:00 | FL | ORLANDO SANFORD IN…| APPROA… SPIRIT AIRLINES | A-319 | 05/10/15 00:00:00 | IL | CHICAGO O'HARE INT…| CLIMB EXPRESSJET AIRLINES| EMB-145 | 05/11/15 00:00:00 | AL | BIRMINGHAM-SHUTTLE…| LANDIN… 1› faa-wildlife-strikes| user_macros | saul.pw/VisiData v2.1 | opening datasets/ 73448 rows
Tell VisiData that HEIGHT is an integer column¶
Navigate to the HEIGHT column, and press #.
You should see something like this:
<STATE | AIRPORT | PHASE_OF_FLT | HEIGHT#| SPEED | SPECIES | BIRDS_STRUCK | > FL | VERO BEACH MUNICIP…| APPROACH | !| | Unknown bird | 1 | … AK | KENAI MUNICIPAL AR…| APPROACH | !| | Unknown bird | 1 | … TX | DAVID WAYNE HOOKS …| | !| | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Unknown bird | 1 | … FL | POMPANO BEACH AIRP…| LANDING ROLL | 0 | | Unknown bird | 1 | VI | HENRY E ROHLSEN AR…| | !| | Unknown bird | 1 | … TX | SAN ANTONIO INTL | APPROACH | !| | Unknown bird | 1 | … TX | LONE STAR EXECUTIV…| DEPARTURE | !| | Unknown bird | 1 | … FL | TAMPA INTL | APPROACH | 6000 | | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Owls | 1 | … FL | OPA-LOCKA EXECUTIV…| APPROACH | !| | Hawks | 1 | … CA | NORMAN Y. MINETA S…| | !| | Gulls | 1 | … FL | FORT LAUDERDALE/HO…| APPROACH | 1500 | | Unknown bird - sma…| 1 | … AR | FORT SMITH REGIONA…| CLIMB | !| | Unknown bird - sma…| 1 | … AR | BILL AND HILLARY …| LANDING ROLL | 0 | | Unknown bird - sma…| 1 | … | UNKNOWN | En Route | !| | Unknown bird | 1 | … CA | METRO OAKLAND INTL | | !| | Unknown bird | 1 | … UT | SALT LAKE CITY INTL| | !| | Unknown bird | 1 | … TX | GEORGE BUSH INTERC…| CLIMB | !| | Unknown bird | 1 | … FL | ORLANDO SANFORD IN…| APPROACH | !| | Unknown bird | 1 | … IL | CHICAGO O'HARE INT…| CLIMB | 12000 | | Unknown bird | 1 | … AL | BIRMINGHAM-SHUTTLE…| LANDING ROLL | 0 | | Unknown bird - sma…| 1 | … 1› faa-wildlife-strikes| # type-int 73448 rows
As you can see, many of the reported collisions are missing height data, or appear to have occurred on the ground (i.e., HEIGHT == 0).
So, let’s focus only on collsions reported to have occurred above the ground.
Select only rows where HEIGHT is greater than zero¶
To do that, we’ll use z|, VisiData’s “select by expression” command. Type z| and then, at the prompt, type HEIGHT > 0. You should see something like this:
<STATE | AIRPORT | PHASE_OF_FLT | HEIGHT#| SPEED | SPECIES | BIRDS_STRUCK | > FL | VERO BEACH MUNICIP…| APPROACH | !| | Unknown bird | 1 | … AK | KENAI MUNICIPAL AR…| APPROACH | !| | Unknown bird | 1 | … TX | DAVID WAYNE HOOKS …| | !| | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Unknown bird | 1 | … FL | POMPANO BEACH AIRP…| LANDING ROLL | 0 | | Unknown bird | 1 | VI | HENRY E ROHLSEN AR…| | !| | Unknown bird | 1 | … TX | SAN ANTONIO INTL | APPROACH | !| | Unknown bird | 1 | … TX | LONE STAR EXECUTIV…| DEPARTURE | !| | Unknown bird | 1 | … FL | TAMPA INTL | APPROACH | 6000 | | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Owls | 1 | … FL | OPA-LOCKA EXECUTIV…| APPROACH | !| | Hawks | 1 | … CA | NORMAN Y. MINETA S…| | !| | Gulls | 1 | … FL | FORT LAUDERDALE/HO…| APPROACH | 1500 | | Unknown bird - sma…| 1 | … AR | FORT SMITH REGIONA…| CLIMB | !| | Unknown bird - sma…| 1 | … AR | BILL AND HILLARY …| LANDING ROLL | 0 | | Unknown bird - sma…| 1 | … | UNKNOWN | En Route | !| | Unknown bird | 1 | … CA | METRO OAKLAND INTL | | !| | Unknown bird | 1 | … UT | SALT LAKE CITY INTL| | !| | Unknown bird | 1 | … TX | GEORGE BUSH INTERC…| CLIMB | !| | Unknown bird | 1 | … FL | ORLANDO SANFORD IN…| APPROACH | !| | Unknown bird | 1 | … IL | CHICAGO O'HARE INT…| CLIMB | 12000 | | Unknown bird | 1 | … AL | BIRMINGHAM-SHUTTLE…| LANDING ROLL | 0 | | Unknown bird - sma…| 1 | … select by expr: HEIGHT > 0 z| select-expr 73448 rows
Then press Enter. Now you you should see the above-ground collisions selected:
<STATE | AIRPORT | PHASE_OF_FLT | HEIGHT#| SPEED | SPECIES | BIRDS_STRUCK | > FL | VERO BEACH MUNICIP…| APPROACH | !| | Unknown bird | 1 | … AK | KENAI MUNICIPAL AR…| APPROACH | !| | Unknown bird | 1 | … TX | DAVID WAYNE HOOKS …| | !| | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Unknown bird | 1 | … FL | POMPANO BEACH AIRP…| LANDING ROLL | 0 | | Unknown bird | 1 | VI | HENRY E ROHLSEN AR…| | !| | Unknown bird | 1 | … TX | SAN ANTONIO INTL | APPROACH | !| | Unknown bird | 1 | … TX | LONE STAR EXECUTIV…| DEPARTURE | !| | Unknown bird | 1 | … •FL | TAMPA INTL | APPROACH | 6000 | | Unknown bird | 1 | … MO | LAMBERT-ST LOUIS I…| APPROACH | !| | Owls | 1 | … FL | OPA-LOCKA EXECUTIV…| APPROACH | !| | Hawks | 1 | … CA | NORMAN Y. MINETA S…| | !| | Gulls | 1 | … •FL | FORT LAUDERDALE/HO…| APPROACH | 1500 | | Unknown bird - sma…| 1 | … AR | FORT SMITH REGIONA…| CLIMB | !| | Unknown bird - sma…| 1 | … AR | BILL AND HILLARY …| LANDING ROLL | 0 | | Unknown bird - sma…| 1 | … | UNKNOWN | En Route | !| | Unknown bird | 1 | … CA | METRO OAKLAND INTL | | !| | Unknown bird | 1 | … UT | SALT LAKE CITY INTL| | !| | Unknown bird | 1 | … TX | GEORGE BUSH INTERC…| CLIMB | !| | Unknown bird | 1 | … FL | ORLANDO SANFORD IN…| APPROACH | !| | Unknown bird | 1 | … •IL | CHICAGO O'HARE INT…| CLIMB | 12000 | | Unknown bird | 1 | … AL | BIRMINGHAM-SHUTTLE…| LANDING ROLL | 0 | | Unknown bird - sma…| 1 | … 1› faa-wildlife-strikes| "HEIGHT > 0" | selected 22883 rows z| select-expr 73448 rows
Create a new sheet with only the selected rows¶
To do so, press ". Once you do that, you should see something like this:
<STATE | AIRPORT | PHASE_OF_FLT | HEIGHT#| SPEED | SPECIES | BIRDS_STRUCK | > FL | TAMPA INTL | APPROACH | 6000 | | Unknown bird | 1 | … FL | FORT LAUDERDALE/HO…| APPROACH | 1500 | | Unknown bird - sma…| 1 | … IL | CHICAGO O'HARE INT…| CLIMB | 12000 | | Unknown bird | 1 | … FL | TAMPA INTL | APPROACH | 50 | | Unknown bird | 1 | … OR | SOUTHWEST OREGON R…| APPROACH | 300 | | Unknown bird | 1 | … ME | PORTLAND INTL JETP…| CLIMB | 1000 | | Unknown bird | 1 | … VA | ROANOKE REGNL ARPT…| APPROACH | 3000 | | Unknown bird | 1 | … FL | MIAMI INTL | APPROACH | 400 | | Unknown bird | 1 | … NY | SYRACUSE HANCOCK I…| APPROACH | 200 | | Unknown bird | 1 | … | UNKNOWN | EN ROUTE | 8000 | | Unknown bird | 1 | … PA | PITTSBURGH INTL AR…| APPROACH | 6000 | | Unknown bird | 1 | … MD | BALTIMORE/WASH INT…| APPROACH | 1000 | | Unknown bird | 1 | … NC | RALEIGH-DURHAM INTL| APPROACH | 1000 | | Unknown bird | 1 | … MI | WILLOW RUN ARPT | APPROACH | 3000 | | Unknown bird - lar…| 1 | … AK | TED STEVENS ANCHOR…| APPROACH | 100 | | Unknown bird | 1 | … | UNKNOWN | LOCAL | 80 | | Unknown bird | 1 | … DC | RONALD REAGAN WASH…| APPROACH | 100 | | Unknown bird | 1 | … AL | BIRMINGHAM-SHUTTLE…| APPROACH | 1100 | | Unknown bird | 1 | … NY | LA GUARDIA ARPT | CLIMB | 400 | | Unknown bird | 1 | … CA | JOHN WAYNE-ORANGE …| APPROACH | 1700 | | Unknown bird - med…| 1 | … FL | TAMPA INTL | APPROACH | 1000 | | Unknown bird | 1 | … NE | EPPLEY AIRFIELD | | 3000 | | Unknown bird - med…| 1 | … 2› faa-wildlife-strikes_selectedref| " 22883 rows
Get the average collision height for each species¶
This is a two-step process. First, navigate to the HEIGHT column, and press + to add an aggregator. At the prompt at the bottom of the screen, type mean …
<STATE | AIRPORT | PHASE_OF_FLT | HEIGHT#| SPEED | SPECIES | BIRDS_STRUCK | > FL | TAMPA INTL | APPROACH | 6000 | | Unknown bird | 1 | … FL | FORT LAUDERDALE/HO…| APPROACH | 1500 | | Unknown bird - sma…| 1 | … IL | CHICAGO O'HARE INT…| CLIMB | 12000 | | Unknown bird | 1 | … FL | TAMPA INTL | APPROACH | 50 | | Unknown bird | 1 | … OR | SOUTHWEST OREGON R…| APPROACH | 300 | | Unknown bird | 1 | … ME | PORTLAND INTL JETP…| CLIMB | 1000 | | Unknown bird | 1 | … VA | ROANOKE REGNL ARPT…| APPROACH | 3000 | | Unknown bird | 1 | … FL | MIAMI INTL | APPROACH | 400 | | Unknown bird | 1 | … NY | SYRACUSE HANCOCK I…| APPROACH | 200 | | Unknown bird | 1 | … | UNKNOWN | EN ROUTE | 8000 | | Unknown bird | 1 | … PA | PITTSBURGH INTL AR…| APPROACH | 6000 | | Unknown bird | 1 | … MD | BALTIMORE/WASH INT…| APPROACH | 1000 | | Unknown bird | 1 | … NC | RALEIGH-DURHAM INTL| APPROACH | 1000 | | Unknown bird | 1 | … MI | WILLOW RUN ARPT | APPROACH | 3000 | | Unknown bird - lar…| 1 | … AK | TED STEVENS ANCHOR…| APPROACH | 100 | | Unknown bird | 1 | … | UNKNOWN | LOCAL | 80 | | Unknown bird | 1 | … 2› faa-wildlife-strikes_selectedref| aggregate-col 22883 rows key ║ desc ║ min ║ minimum value ║ max ║ maximum value ║ avg ║ arithmetic mean of values ║ mean ║ arithmetic mean of values ║ +› choices| "mean" / search-col 16 choices
… and then press Enter.
Note
Adding an aggregator doesn’t change the appearance of the sheet. But if you want to confirm that you’ve added an aggregator correctly, you can press Shift-C to open up the Columns Sheet, and look at the aggregators field.
Then, to get the average for each animal, navigate to the SPECIES column, and press Shift-F to create a frequency sheet. It should look something like this:
SPECIES ║↓count♯| HEIGHT_mean%║ Unknown bird - sma…║ 8807 | 1408.82 ║ Unknown bird - med…║ 3056 | 2260.49 ║ Unknown bird ║ 2098 | 2253.93 ║ Unknown bird - lar…║ 693 | 2691.92 ║ Gulls ║ 550 | 696.03 ║ Barn swallow ║ 385 | 191.71 ║ Mourning dove ║ 312 | 229.82 ║ European starling ║ 277 | 181.48 ║ Sparrows ║ 252 | 554.77 ║ Rock pigeon ║ 219 | 376.45 ║ Perching birds (y) ║ 211 | 2018.73 ║ Canada goose ║ 208 | 848.81 ║ Killdeer ║ 199 | 189.23 ║ Red-tailed hawk ║ 190 | 370.29 ║ American robin ║ 183 | 2545.00 ║ Turkey vulture ║ 178 | 1119.58 ║ American kestrel ║ 159 | 202.04 ║ Horned lark ║ 159 | 290.84 ║ Chimney swift ║ 156 | 928.85 ║ Cliff swallow ║ 135 | 270.13 ║ Mallard ║ 133 | 1173.14 ║ Hawks ║ 131 | 770.13 ║ 3› faa-wildlife-strikes_selectedref_SPECIES_freq| F 420 bins
Sort the species by collision height¶
By default, frequency tables are sorted by the raw count of matching rows. To sort by the highest average collision height, navigate to the mean_HEIGHT column, and press ].
You should see something like this:
SPECIES ║ count♯|↓HEIGHT_mean%║ Eared grebe ║ 1 | 20000.00 ║ Long-billed curlew ║ 1 | 15000.00 ║ Whimbrel ║ 1 | 12000.00 ║ Lesser yellowlegs ║ 4 | 9925.00 ║ Dusky flycatcher ║ 1 | 9500.00 ║ Sanderling ║ 1 | 9500.00 ║ Red-necked phalaro…║ 4 | 8525.00 ║ Clark's grebe ║ 1 | 8500.00 ║ Northern saw-whet …║ 1 | 8000.00 ║ Grebes ║ 1 | 8000.00 ║ Spotted sandpiper ║ 1 | 7700.00 ║ Least flycatcher ║ 3 | 7566.67 ║ Lark sparrow ║ 2 | 7400.00 ║ Greater yellowlegs ║ 3 | 7333.33 ║ Pacific-slope flyc…║ 3 | 7000.00 ║ Seminole bat ║ 1 | 7000.00 ║ Solitary sandpiper ║ 2 | 6250.00 ║ Pocketed free-tail…║ 1 | 6000.00 ║ Red-naped sapsucker║ 2 | 5750.00 ║ Cinnamon teal ║ 1 | 5600.00 ║ Horned grebe ║ 2 | 5400.00 ║ Willow flycatcher ║ 2 | 5200.00 ║ 3› faa-wildlife-strikes_selectedref_SPECIES_freq| ] sort-desc 420 bins
Hrrrm, it seems like a lot of these species show up just a few times — too few to be particularly informative. Let’s do something about that.
Limit the results to relatively common species¶
This step will seem familiar; it’s a lot like how we selected only above-the-ground collisions.
First, type z| to raise the “select by expression” prompt. Then, type count >= 20:
SPECIES ║ count♯|↓HEIGHT_mean%║ Eared grebe ║ 1 | 20000.00 ║ Long-billed curlew ║ 1 | 15000.00 ║ Whimbrel ║ 1 | 12000.00 ║ Lesser yellowlegs ║ 4 | 9925.00 ║ Dusky flycatcher ║ 1 | 9500.00 ║ Sanderling ║ 1 | 9500.00 ║ Red-necked phalaro…║ 4 | 8525.00 ║ Clark's grebe ║ 1 | 8500.00 ║ Northern saw-whet …║ 1 | 8000.00 ║ Grebes ║ 1 | 8000.00 ║ Spotted sandpiper ║ 1 | 7700.00 ║ Least flycatcher ║ 3 | 7566.67 ║ Lark sparrow ║ 2 | 7400.00 ║ Greater yellowlegs ║ 3 | 7333.33 ║ Pacific-slope flyc…║ 3 | 7000.00 ║ Seminole bat ║ 1 | 7000.00 ║ Solitary sandpiper ║ 2 | 6250.00 ║ Pocketed free-tail…║ 1 | 6000.00 ║ Red-naped sapsucker║ 2 | 5750.00 ║ Cinnamon teal ║ 1 | 5600.00 ║ Horned grebe ║ 2 | 5400.00 ║ Willow flycatcher ║ 2 | 5200.00 ║ select by expr: count >= 20 z| select-expr 420 bins
Next, press Enter to complete the action. Because there are no high-count species in the visible part of the sheet, you won’t notice much of effect at first; just a confirmation in the status bar at the bottom:
Horned grebe ║ 2 | 5400.00 ║ Willow flycatcher ║ 2 | 5200.00 ║ 3› faa-wildlife-strikes_selectedref_SPECIES_freq| "count >= 20" | s z| select-expr 420 bins
Now press " to create a new sheet containing only the selected rows. Tada!:
SPECIES ║ count♯| HEIGHT_mean%║ Yellow-bellied sap…║ 24 | 3585.42 ║ Swainson's thrush ║ 56 | 3454.46 ║ Snow goose ║ 34 | 3258.82 ║ Hermit thrush ║ 49 | 2770.10 ║ Red-eyed vireo ║ 39 | 2768.72 ║ Unknown bird - lar…║ 693 | 2691.92 ║ Northern shoveler ║ 23 | 2602.17 ║ American robin ║ 183 | 2545.00 ║ White-throated spa…║ 56 | 2426.79 ║ Greater white-fron…║ 32 | 2335.94 ║ Unknown bird - med…║ 3056 | 2260.49 ║ Unknown bird ║ 2098 | 2253.93 ║ Dark-eyed junco ║ 39 | 2031.28 ║ Perching birds (y) ║ 211 | 2018.73 ║ Northern pintail ║ 64 | 1980.19 ║ American coot ║ 53 | 1965.66 ║ Unknown bird or bat║ 43 | 1892.88 ║ Red bat ║ 20 | 1750.00 ║ Geese ║ 56 | 1696.88 ║ Gadwall ║ 25 | 1675.80 ║ Gray catbird ║ 37 | 1573.27 ║ New World Vultures ║ 60 | 1541.50 ║ 4› faa-wildlife-strikes_selectedref_SPECIES_freq_selectedref| " 75 bins
Finally, type g_ to widen all the columns, so that you can read all the names:
SPECIES ║ count♯| HEIGHT_mean%║ Yellow-bellied sapsucker ║ 24 | 3585.42 ║ Swainson's thrush ║ 56 | 3454.46 ║ Snow goose ║ 34 | 3258.82 ║ Hermit thrush ║ 49 | 2770.10 ║ Red-eyed vireo ║ 39 | 2768.72 ║ Unknown bird - large ║ 693 | 2691.92 ║ Northern shoveler ║ 23 | 2602.17 ║ American robin ║ 183 | 2545.00 ║ White-throated sparrow ║ 56 | 2426.79 ║ Greater white-fronted goose ║ 32 | 2335.94 ║ Unknown bird - medium ║ 3056 | 2260.49 ║ Unknown bird ║ 2098 | 2253.93 ║ Dark-eyed junco ║ 39 | 2031.28 ║ Perching birds (y) ║ 211 | 2018.73 ║ Northern pintail ║ 64 | 1980.19 ║ American coot ║ 53 | 1965.66 ║ Unknown bird or bat ║ 43 | 1892.88 ║ Red bat ║ 20 | 1750.00 ║ Geese ║ 56 | 1696.88 ║ Gadwall ║ 25 | 1675.80 ║ Gray catbird ║ 37 | 1573.27 ║ New World Vultures ║ 60 | 1541.50 ║ 4› faa-wildlife-strikes_selectedref_SPECIES_freq_selectedref| g_ resize-cols-max 75 bins